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Uninhabited  Military  Vehicles  (UMVs):  Human 
Factors  Issues  in  Augmenting  the  Force 

(RTO-TR-HFM-078) 

Executive  Summary 

A  number  of  NATO  countries  are  now  using  Uninhabited  Military  Vehicles  (UMVs)  to  augment  their 
manned  forces,  especially  in  performing  tasks  which  are  dull,  dirty,  or  dangerous.  Force  augmentation  issues 
relevant  to  the  human  operator  exist  on  several  levels,  including  individual  UMV  control  station  design, 
vehicle  interoperability,  and  integration  of  UMVs  with  manned  systems.  Human  interface  issues  associated 
with  individual  UMV  control  station  design  include  guaranteeing  appropriate  situational  awareness  for  the 
task,  minimizing  adverse  effects  of  lengthy  system  time  delays,  establishing  an  optimum  ratio  of  operators  to 
vehicles,  and  providing  effective  information  presentation  and  control  strategies.  UMV  interoperability 
requires  development  of  a  standard  set  of  control  station  design  specifications/procedures/heuristics  to  cover 
the  range  of  potential  UMV  operators/applications  across  military  services  and  countries.  Finally,  for  UMVs 
to  be  successful,  they  must  be  successfully  integrated  with  manned  systems  so  as  to  enhance  the  strength  of 
the  overall  force.  Human  factors  considerations  in  this  area  include  how  manned  systems  should  best 
collaborate  with  UMVs,  deconfliction  concerns,  and  command  and  control  issues. 

Given  the  current  progress  of  technological  developments  and  operational  concepts  regarding  UMVs, 
a  strong  and  combined  effort  of  NATO-countries  is  essential  to  resolve  the  unique  human-system  issues 
associated  with  augmenting  the  existing  force  with  these  vehicles.  New  principles  for  supporting  the 
operator  in  such  scenarios,  and  for  collaboration  between  multiple  operators  in  vehicles,  need  to  be 
developed  and  evaluated. 

The  goal  of  this  report  is  to  assemble  pertinent  information  concerning  the  factors  that  will  increase 
NATO’s  successful  operations  through  effective  force  augmentation  with  UMVs.  By  bringing  this 
information  together  in  one  report,  decision-makers  and  scientists  will  be  able  to  consider  the  numerous 
factors  that  have  to  be  taken  into  account  for  operators  to  successfully  work  with  UMVs.  These  different 
factors  are  discussed  in  the  following  chapters:  Scenarios  and  Military  Relevance,  Theoretical 
Frameworks,  System  of  Systems,  Artificial  Cognition  and  Cooperative  Automation,  Controls  and 
Displays,  and  Human  Automation  Integration.  The  diversity  of  the  subjects  illustrates  the  challenge  of 
successfully  integrating  UMVs  into  NATO  forces. 

The  paragraphs  below  give  a  short  description  of  the  key  factors  needed  for  successful  integration. 

Scenarios  and  Military  Relevance.  Modern  warfare  requires  military  capabilities  that  respond  to  the 
threat  of  conventional  hostile  force  and  to  the  challenges  of  asymmetric  conflicts,  where  political  and 
military  success  relies  on  effects-based  targeting  and  operations.  Important  questions  remain  about  what 
realistic  effects  can  be  expected  to  be  achieved  by  UMVs  in  the  uncertain,  ambiguous  and  non-linear 
battle-space  of  the  future,  including  how  international  law  will  interpret  robotic  warfare  in  the  future. 
Since  UMV  technologies  are  expected  to  actually  reduce  human  involvement  in  some  tasks,  it  is  not  self 
evident  why  HF  issues  should  warrant  raised  attention.  Thus,  the  reasons  for  raising  HF  concerns  are  in 
need  of  review. 

Theoretical  Frameworks.  Frameworks  have  been  used  to  guide  the  design  of  technology,  procedures, 
systems,  and  systems  of  systems.  UMV  systems  will  also  require  theoretical  frameworks  to  inform  the 
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design  process.  Most  of  the  frameworks  used  in  traditional  manned  systems  can  be  applied  to  uninhabited 
systems.  However,  revisiting  the  theoretical  frameworks  discussion  allows  us  to  highlight  aspects  of  the 
frameworks  that  are  directly  applicable  to  optimizing  operator/vehicle  ratios  and  interoperability  of 
uninhabited  systems.  In  the  investigation  we  may  also  find  an  emerging  theory  or  framework  that  is 
unique  to  UMV  systems. 

System  of  Systems.  New  system  architectures  designed  for  interoperability  are  being  developed  to 
integrate  multiple  platforms  into  a  common  mission  control  element,  giving  the  war-fighter  access  to  a 
large  volume  of  real-time  information.  Resolving  issues  associated  with  connectivity,  knowledge  and 
action  consistency,  and  transfer  of  control  have  taken  center  stage  along  with  traditional  Human  Factors 
issues  related  to  information  management,  information  processing,  decision  aiding,  levels  of  autonomy, 
command  and  control,  manpower  and  skills,  and  training.  To  be  successful,  these  issues  must  be  addressed 
during  the  early  stages  of  systems  engineering  to  ensure  proper  human-centered  development  of  UMV 
systems  within  a  system-of-systems  architecture. 

Artificial  Cognition  and  Cooperative  Automation.  This  section  describes  a  systems  engineering 
framework  which  facilitates  the  definition  of  required  automation  functions  and  addresses  the  need  for 
intelligent  automation  in  order  to  guide  uninhabited  military  vehicles.  This  framework  is  primarily  based 
upon  considerations  of  human  performance  and  cognitive  engineering.  Taking  a  general  and  well  accepted 
model  of  human  cognition  as  a  starting  point,  the  issues  of  intelligent  machine  performance  in  decision¬ 
making,  human-machine  teamwork  in  a  distributed  system,  and  level  of  automation/system  autonomy, 
in  terms  of  allocation  of  functions  and  responsibilities,  are  discussed  at  a  conceptual  depth. 

Controls  and  Displays.  As  application  of  automation  technology  to  the  UMV  domain  increases, 
the  potential  for  unintentional  side  effects  associated  with  highly  automated  systems  needs  to  be 
considered.  These  include  rapid  and  significant  fluctuations  in  operator  workload,  loss  of  situational 
awareness,  lack  of  perceived  reliability,  complacency,  skill  loss,  and  operator  performance  decrements. 
To  truly  have  a  human-centered  approach  when  incorporating  automation  in  UMV  applications,  it  is 
necessary  to  employ  well  designed  controls  and  displays  -  controls  that  enable  efficient  task  completion 
and  displays  that  keep  the  remote  operator  well-informed.  A  goal  of  the  interface  is  to  recreate  some  of  the 
cues  available  in  the  world.  Indeed,  it  could  be  said  that  the  primary  goal  of  the  interface  should  be  to 
reproduce  the  perceptual  cues  that  are  used  by  the  operators  in  the  real  environment  to  perform  the  task. 

Human  Automation  Integration.  To  a  large  extent,  operators  will  interact  with  UMVs  using  supervisory 
control.  In  addition,  UMVs  will  contain  multi-agent  associate  software  that  enables  adaptive  and 
adjustable  automation  between  the  operator  and  the  vehicle.  By  utilizing  dynamic  function  allocation  and 
various  levels  of  automation,  an  architecture  can  be  designed  to  create  an  efficient  team  between  the 
operator  and  the  vehicle.  Research  conducted  by  NATO  team  members  has  already  demonstrated  the 
employment  of  user  knowledge  acquisition  to  produce  a  practical,  communicable  set  of  assisted  levels 
(At  Call,  Advisory,  In  Support,  Direct  Support)  for  variable  and  adaptive  decision  support/automation, 
supporting  situational  assessment,  decision  making  and  action. 
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Vehicules  militaires  sans  pilote  (UMV) : 
Questions  relatives  aux  facteurs  humains 
lies  a  V  augmentation  des  forces 
(RTO-TR-HFM-078) 


Synthese 


Un  certain  nombre  de  nations  de  l’OTAN  utilisent  desormais  des  Vehicules  militaires  sans  pilote  (UMV) 
tout  particulierement  pour  l’accomplissement  de  taches  ennuyeuses,  salissantes  ou  dangereuses,  dans  le  but 
d’augmenter  leurs  moyens  humains.  Les  questions,  liees  a  l’augmentation  des  forces,  qui  concement 
l’operateur  humain  existent  a  plusieurs  niveaux,  parmi  lesquels  la  conception  de  postes  de  commande 
d’UMV  individuels,  1’ interoperability  des  vehicules,  et  l’integration  d’UMV  avec  des  systemes  pilotes. 
Les  notions  d’ interface  humaine  associees  a  la  conception  de  postes  de  commande  d’UMV  individuels 
garantissent  une  connaissance  de  la  situation  adaptee  a  la  tache,  la  limitation  des  effets  indesirables  de  la 
lenteur  des  systemes,  la  determination  d’un  rapport  optimal  operateurs/vehicules,  et  la  foumiture  de 
strategies  de  controle  et  d’une  presentation  d’ informations  efficaces.  L’ interoperability  des  UMV  necessite  le 
developpement  d’une  serie  normative  de  specifications/de  procedures/d’heuristique  de  conception  de  postes 
de  commande,  afin  de  couvrir  la  gamme  des  operateurs/applications  potentiels  des  UMV  parmi  les  services 
militaires  et  les  nations.  Enfin,  pour  que  les  UMV  reussissent,  ils  doivent  etre  integres  avec  succes  aux 
systemes  avec  pilote,  de  fa$on  a  augmenter  la  puissance  de  l’ensemble  de  la  force.  Les  considerations  liees 
aux  facteurs  humains  en  ce  domaine  incluent  la  meilleure  maniere,  pour  les  systemes  pilotes,  de  collaborer 
avec  les  UMV,  les  questions  de  deconfliction,  et  les  problemes  de  commandement  et  de  controle. 

Etant  donne  les  progres  actuels  des  developpements  technologiques  et  des  concepts  operationnels  dans  le 
domaine  des  UMV,  un  effort  soutenu  et  combine  de  la  part  des  nations  de  l’OTAN  est  primordial  pour 
resoudre  les  problemes  particuliers  homme-systeme  associes  a  V  augmentation  des  forces  existantes  a 
l’aide  de  ces  vehicules.  De  nouveaux  principes  pour  soutenir  l’operateur  dans  de  tels  scenarios,  et  pour  la 
collaboration  entre  des  operateurs  multiples  dans  les  vehicules,  doivent  etre  developpes  et  evalues. 

L’objectif  de  ce  rapport  est  de  rassembler  des  informations  pertinentes  sur  les  facteurs  qui  augmenteront  le 
nombre  d’operations  reussies  de  l’OTAN  grace  a  l’accroissement  effectif  des  forces  par  les  UMV. 
En  reunissant  ces  informations  en  un  seul  et  meme  rapport,  les  decideurs  et  les  scientifiques  pourront 
reflectin'  aux  nombreux  facteurs  a  prendre  en  consideration  pour  que  les  operateurs  puissent  travailler  avec 
succes  avec  les  UMV.  Ces  differents  facteurs  sont  examines  dans  les  chapitres  suivants  :  Scenarios  et 
pertinence  militaire,  Cadres  theoriques,  Systeme  de  systemes,  Cognition  artificielle  et  automatisation 
cooperative,  Commandes  et  affichages,  et  Integration  homme-automatisation.  La  diversity  des  sujets 
illustre  le  defl  que  represente  l’integration  reussie  des  UMV  dans  les  forces  de  l’OTAN. 

Les  paragraphes  ci-dessous  decrivent  brievement  les  facteurs  cles  necessaries  a  une  integration  reussie. 

Scenarios  et  pertinence  militaire.  La  guerre  moderne  necessite  des  moyens  militaires  qui  repondent  a  la 
menace  de  forces  hostiles  conventionnelles  et  aux  defls  des  conflits  asymetriques,  ou  le  succes  politique  et 
militaire  repose  sur  le  choix  des  objectifs  et  des  moyens  de  traitement  et  sur  les  operations  en  fonction  des 
effets.  Des  questions  importantes  demeurent  quant  aux  effets  realistes  que  l’on  peut  esperer  obtenir  avec 
les  UMV  sur  le  champ  de  bataille  incertain,  ambigu  et  non  lineaire  du  futur,  y  compris  la  maniere  dont  le 
droit  international  traitera  la  guerre  robotique  dans  le  futur.  Dans  la  mesure  ou  Ton  s’ attend  a  ce  que  les 
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technologies  des  UMV  reduisent  en  fait  l’engagement  humain  pour  certaines  taches,  il  n’est  pas  evident 
que  les  questions  de  FH  justifient  une  attention  soutenue.  En  consequence,  les  raisons  de  soulever  des 
problemes  de  FH  doivent  etre  revisees. 

Cadres  theoriques.  Des  cadres  ont  ete  utilises  pour  guider  la  conception  de  technologies,  de  procedures,  de 
systemes,  et  de  systemes  de  systemes.  Les  systemes  d’UMV  necessiteront  egalement  des  cadres  theoriques 
pour  renseigner  le  processus  de  conception.  La  plupart  des  cadres  utilises  dans  les  systemes  pilotes 
traditionnels  peuvent  etre  appliques  aux  systemes  sans  pilote.  Toutefois,  revisiter  la  question  des  cadres 
theoriques  nous  permet  de  mettre  en  lumiere  des  aspects  de  ces  cadres  qui  sont  directement  applicables  a 
F optimisation  des  rapports  operateur/vehicule  et  de  F interoperability  des  systemes  sans  pilote.  Au  cours  de 
l’etude,  nous  pouvons  aussi  decouvrir  une  theorie  emergente  ou  un  cadre  particular  aux  systemes  d’UMV. 

Systeme  de  systemes.  De  nouvelles  architectures  de  systemes  congues  pour  F  interoperability  sont  en  train 
d’etre  developpees  pour  integrer  les  plates-formes  multiples  dans  un  organe  de  commande  de  mission 
commun,  donnant  au  soldat  la  possibility  d’acceder  a  un  volume  important  d’ informations  en  temps  reel. 
La  resolution  de  problemes  lies  a  la  connectivity,  la  coherence  des  connaissances  et  des  actions,  et  le 
transfert  de  commande  occupent  desormais  le  devant  de  la  scene,  aux  cotes  des  problemes  traditionnels  de 
Facteurs  humains  relatifs  a  la  gestion  et  au  traitement  des  informations,  a  l’aide  a  la  decision,  aux  niveaux 
d’autonomie,  au  commandement  et  au  controle,  a  la  main  d’oeuvre  et  aux  aptitudes,  et  a  la  formation. 
Pour  reussir,  ces  questions  doivent  etre  abordees  au  cours  des  premiers  stades  d’ingenierie  des  systemes 
afin  de  garantir  un  developpement  adapte,  centre  sur  l’humain,  des  systemes  d’UMV  au  sein  d’une 
architecture  de  systeme  de  systemes. 

Cognition  artificielle  et  automatisation  cooperative.  Cette  section  decrit  un  cadre  d’ingenierie  de 
systemes  qui  facilite  la  definition  des  fonctions  d’ automatisation  requises  et  aborde  la  necessity  d’une 
automatisation  intelligente  pour  guider  les  vehicules  militaires  sans  pilote.  Ce  cadre  est  principalement 
fonde  sur  des  considerations  de  performances  humaines  et  d’ingenierie  cognitive.  En  prenant  comme  point 
de  depart  un  modele  general  et  largement  admis  de  cognition  humaine,  les  problemes  de  performances  des 
machines  intelligentes  dans  la  prise  de  decisions,  le  travail  d’equipe  homme-machine  dans  un  systeme 
reparti,  et  le  niveau  d’automatisation/d’autonomie  du  systeme,  en  termes  d’attribution  des  fonctions  et  des 
responsabilites,  sont  examines  a  un  niveau  conceptuel. 

Commandes  et  affichages.  Avec  l’augmentation  de  l’application  de  la  technologie  d’ automatisation  au 
domaine  des  UMV,  la  possibility  d’effets  indesirables  non  intentionnels  lies  aux  systemes  fortement 
automatises  doit  etre  prise  en  compte.  Cela  comprend  les  fluctuations  rapides  et  importantes  de  la  charge 
de  travail  de  l’operateur,  la  perte  de  la  connaissance  de  la  situation,  l’absence  de  fiabilite  pergue,  l’exces 
de  confiance,  la  perte  d’aptitudes,  et  les  diminutions  de  performances  de  l’operateur.  Afin  d’avoir  une 
approche  reellement  centree  sur  l’humain  lors  de  l’integration  de  1’ automatisation  dans  les  applications 
UMV,  il  est  necessaire  d’utiliser  des  commandes  et  des  affichages  bien  congus  -  des  commandes  qui 
permettent  l’execution  efficace  des  taches,  et  des  affichages  qui  maintiennent  l’operateur  a  distance  bien 
informe.  L’un  des  buts  de  l’interface  est  de  recreer  certains  des  reperes  disponibles  dans  le  monde  reel. 
En  effet,  on  pourrait  dire  que  l’objectif  principal  de  l’interface  devrait  etre  de  reproduire  les  reperes 
sensoriels  utilises  par  les  operateurs  en  environnement  reel  pour  l’accomplissement  de  la  tache. 

Integration  homme-automatisation.  Dans  une  large  mesure,  les  operateurs  interagiront  avec  les  UMV  en 
utilisant  un  dispositif  de  surveillance.  En  outre,  les  UMV  comprendront  des  logiciels  associes  multi-agents 
qui  permettent  1’ automatisation  adaptative  et  ajustable  entre  l’operateur  et  le  vehicule.  Grace  a  l’attribution 
de  fonctions  dynamiques  et  a  divers  niveaux  d’ automatisation,  une  architecture  peut  etre  congue  en  vue  de 
faire  de  l’operateur  et  du  vehicule  une  equipe  efficace.  Les  recherches  menees  par  les  membres  de  l’equipe 
de  l’OTAN  ont  deja  decrit  l’utilisation  de  l’acquisition  de  connaissances  de  l’utilisateur  en  vue  de  produire 
une  serie  pratique,  communicable,  de  niveaux  assistes  (sur  demande,  consultatif,  en  appui,  appui  direct) 
pour  l’aide  a  la  decision/1’ automatisation  variables  et  adaptatives,  soutenant  revaluation  de  la  situation,  la 
prise  de  decision  et  Faction. 
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Chapter  1  -  INTRODUCTION 

Chapter  Lead:  R.  Taylor 
Contributors:  J.  Reising,  R.  Taylor 


The  terms  of  Reference  (TOR)  for  the  Task  Group  (TG)  lists  as  its  objective  to  seek  to  augment  the  force 
using  uninhabited  military  vehicles  (UMVs)  by  leveraging  the  potential  advantages  of  UMVs  to  act  as  force 
multipliers.  Since  there  are  no  truly  uninhabited  systems  -  operators  will  always  be  in  the  loop  in  some 
fashion  -  human  factors  issues  become  crucial  to  the  successful  operation  of  these  systems.  Force 
multiplication  can  be  achieved  by  addressing  the  human  factors  issues  and  challenges  shown  below. 

•  Collaborative  Work  -  Optimal  Task  Distribution; 

•  Virtual  team  performance; 

•  Manned/Unmanned  collaboration; 

•  Interoperability; 

•  Flexible  level  of  automation; 

•  Optimization  of  operator/vehicle  ratio; 

•  Control  Stations  -  Intelligent  Operator  Support; 

•  Operator  functional  state  assessment; 

•  Intelligent  adaptive  interfaces; 

•  Cognitive  cooperation;  and 

•  Knowledge  management  systems. 

After  NATO/RTO  approval  of  the  TOR,  the  TG  was  formed.  Seven  countries  agreed  to  participate:  Canada, 
France,  Germany,  The  Netherlands,  Sweden,  United  Kingdom  and  United  States. 


1.1  THE  ISSUES 

Following  some  initial  meetings,  a  crucial  symposium  was  held  in  Leiden,  The  Netherlands  to  frame  the 
issues  to  be  addressed  by  the  TG.  The  results  of  the  symposium  led  to  the  following  five  key  issues  which 
form  the  basis  of  discussion  in  the  final  technical  report: 

1)  Theoretical  Frameworks; 

2)  System  of  Systems; 

3)  Cooperative  Automation  and  Computational  Intelligence; 

4)  Controls  and  Displays;  and 

5)  Human-Automation  Integration. 

After  subsequent  meetings,  a  chapter  on  Scenarios  and  Military  Relevance  was  added.  With  the  addition  of 
Introduction  and  Summary  and  Conclusions  chapters,  and  the  modification  of  some  chapter  titles,  the  technical 
report  (TR)  now  contains  the  eight  chapters  listed  below: 
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1)  Introduction 

2)  Scenarios  and  Military  Relevance 

3)  Theoretical  Frameworks 

4)  System  of  Systems 

5)  Artificial  Cognition  and  Cooperative  Automation 

6)  Controls  and  Displays 

7)  Human- Automation  Integration 

8)  Summary  and  Conclusions 

The  objective  of  this  Introduction  is  to  give  an  overview  of  the  key  issues  discussed  in  each  of  these  chapters. 

1.2  SCENARIOS  AND  MILITARY  RELEVANCE 

UMVs  are  enablers  of  military  capability  with  clear  endorsement  at  the  highest  level.  Many  NATO  Nations 
have  active  programmes  to  develop  and  integrate  UMV  systems  into  the  front  line  military  force  mix.  UMVs 
are  most  commonly  characterised  as  dealing  well  with  “3D”  tasks  -  dull,  dirty  and  dangerous.  They  are  used 
extensively  in  intelligence,  surveillance  and  reconnaissance  (ISR),  roles  affording  persistence  in  the  provision 
of  critical  information,  without  risking  lives.  Increasingly,  they  are  being  utilized  for  combat  and  support 
roles.  Important  questions  remain  about  what  realistic  effects  can  be  expected  to  be  achieved  by  UMVs  in  the 
uncertain,  ambiguous  and  non-linear  battle-space  of  the  future,  including  how  international  law  will  interpret 
robotic  warfare  in  the  future. 

UMVs  are  used  extensively  to  gather  information  in  ISR  roles  for  human  interpretation.  ISR  information  is 
inherently  incomplete  and  uncertain.  Fundamentally,  computer-based  information  processing  systems  are 
limited  in  that  they  can  not  comprehend  the  meaning  of  information  in  human  cognitive  terms,  e.g.,  apply 
knowledge,  understand,  feel  truth,  appreciate  implications,  judge  consequences.  Critical  military  judgment  is 
needed  to  interpret  the  meaning  of  ISR  information.  Crucially,  UMVs  can  not  appreciate  the  effects  of  the  use 
of  lethal  force.  This  lack  of  appreciation  of  lethal  force  consequences  is  one  of  the  key  issues  why  human 
factors  are  important  military  relevant  issues  with  “unmanned”  technologies.  An  example  of  this  is  illustrated 
in  the  use  of  autonomous  UAVs.  Autonomy  is  needed  so  that  degraded  communications,  whether  caused  by 
sunspots  or  jamming,  must  not  impair  the  aircraft  functionality  or  the  system’s  ability  to  complete  missions 
within  the  assigned  rules  of  engagement  (ROE).  The  example  ROE  given  is  the  use  of  force  only  if  authorized 
by  the  human  operator.  An  excellent  summarization  of  the  ethical/moral  issues  utilizing  UMVs  in  combat  was 
discussed  by  Air  Chief  Marshall  Sir  Brian  Burridge.  (Reference  29  in  Chapter  2) 

“When  we  go  into  combat,  we  have  got  to  be  sure  what  we  are  doing  is  both  legal  and  moral. 

I  do  not  believe  that,  in  future,  even  though  technology  will  allow  it,  we  will  be  allowed  to 
indulge  in  robotic  warfare.  I  simply  do  not  see  the  international  community  regarding  that  as  an 
appropriate  way  to  fight.  The  notion  of  using  UCAVs  controlled  from  10  time  zones  away  to 
prosecute  a  battle  is  not  something  international  law  of  the  future  will  regard  as  acceptable. 

I  think  the  notion  of  a  person  in  the  loop,  the  notion  of positive  ID,  the  notion  of  someone  feeling 
the  texture  of  what  is  going  on  in  the  battlespace,  is  going  to  be  more  and  more 

prevalent . Overall,  I  think  robotic  warfare  drives  you  away  from  what  I  term  as  emotional 

connectivity  with  the  battlespace.  My  view  is  that  winning  the  hearts  and  minds  battle  with  the 
indigenous  population  requires  this  emotional  connectivity.  ” 
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Note:  In  this  report,  when  referencing  UMVs,  the  term  “uninhabited”  will  be  substituted  for  “unmanned” 
where  appropriate,  in  recognition  of  the  role  of  both  women  and  men  equally  in  serving  our  armed  forces. 


1.3  THEORETICAL  FRAMEWORKS 

Theoretical  frameworks  have  been  used  to  guide  the  design  of  technology,  procedures,  systems,  and  systems 
of  systems.  UMV  systems  will  also  require  theoretical  frameworks  to  inform  the  design  process.  Most  of  the 
frameworks  used  in  traditional  manned  systems  can  be  applied  to  uninhabited  systems.  However,  revisiting 
the  theoretical  frameworks  discussion  allows  us  to  highlight  aspects  of  the  frameworks  that  are  directly 
applicable  to  optimizing  operator/vehicle  ratios  and  interoperability  of  uninhabited  systems.  In  the  investigation 
we  may  also  find  an  emerging  theory  or  framework  that  is  unique  to  UMV  systems. 

The  place  for  theory  in  design  is  as  follows: 

•  Theory  can  be  the  starting  point  for  design; 

•  Theory  may  identify  the  critical  design  decisions; 

•  Theory  allows  for  a  common  taxonomy  within  and  across  systems; 

•  Theory  helps  track  and  maintain  the  aim  throughout  the  system  life  cycle; 

•  Theory  helps  design  system  verification  and  validation;  and 

•  Theory  helps  generate  measures  of  effectiveness. 

There  are  a  number  of  theoretical  frameworks  that  address  operator/vehicle  optimization  and  interoperability. 
Theoretical  frameworks  developed  for  operator-manned  vehicle  interaction  can  be  applied  to  uninhabited 
systems  when  it  comes  to  basic  ergonomics,  workstation  design,  task  analysis,  workload  and  situational 
awareness.  In  most  cases,  human-machine  interaction  theories  apply  regardless  if  humans  are  inside  or  outside 
of  the  vehicle,  although  ego-  versus  exo-centric  frames  of  reference  may  become  an  issue  specific  to  UMVs. 
Human-human  interaction  theories  (i.e.,  social  behaviour)  might  better  describe  operators  who  interact  with 
vehicles  as  a  team.  Thus  human-machine  and  human-human  interaction  theories  are  reasonable  starting  points 
for  exploring  operator/vehicle  and  interoperability  optimization. 

The  choice  of  a  framework  for  analysing  and  designing  UMV  systems  may  depend  on  the  proposed  solution. 
For  example,  if  reducing  the  operator/vehicle  ratio  means  going  from  three  operators  operating  one  vehicle  to 
one  operator  operating  three  vehicles,  then  one  can  imagine  the  requirement  for  intelligent  help  and  levels  of 
automation.  The  theoretical  framework  will  need  to  address  the  following  aspects: 

•  Level  of  automation  and  time  and  cultural  dependencies. 

•  Goal/constraint  level  interactions  instead  of  action  level  interactions. 

•  Self-generating  future  plans. 

•  Environment  and  system  unpredictability. 

•  Trust  and  system  acceptability  and  predictability. 

•  Implications  of  truly  autonomous  (free  will)  systems. 

•  Animation  and  personification  of  machines. 

•  Self-awareness,  environment  awareness,  and  awareness  of  itself  within  its  environment. 
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While  the  theory  may  be  the  starting  point  of  the  design,  aspects  of  the  design  define  the  theoretical 
framework  to  be  applied.  There  is  some  initial  iteration  and  recursion  in  determining  the  theoretical 
framework;  however  this  recursion  should  quickly  converge  so  that  the  design  can  move  forward. 

1.4  SYSTEM  OF  SYSTEMS 

Once  dismissed  as  novel  technology  that  would  never  be  useful  within  a  dynamic  environment,  UMVs  are 
being  developed  in  greater  numbers  and  growing  sophistication  as  the  modem  military  strives  for  greater 
persistence  over  the  battlefield,  more  real-time  intelligence,  and  the  ability  to  strike  heavily  defended  targets. 
New  system  architectures  designed  for  interoperability  are  being  developed  to  integrate  multiple  platforms 
into  a  common  mission  control  element  giving  the  war-fighter  access  to  a  large  volume  of  real-time 
information.  The  end  result  is  an  entire  set  of  new  Human  Factors  related  challenges  facing  developers  to 
ensure  successful  human  systems  integration.  Resolving  issues  associated  with  connectivity,  knowledge  and 
action  consistency,  and  transfer  of  control  have  taken  center  stage  along  with  traditional  Human  Factors 
issues  related  to  information  management,  information  processing,  decision  aiding,  levels  of  autonomy, 
and  command  and  control  (C2). 

As  one  example,  consider  manpower  and  skills,  and  training.  The  transfer  of  skills  and  knowledge,  and  the 
requirement  for  general  skill  and  knowledge  levels  will  contribute  to  force  multiplication  by  drawing  from  an 
existing,  broader  pool  of  people  that  can  operate  UMVs.  There  are  also  new  challenges  connected  with 
embedded  training.  We  clearly  do  not  want  to  compromise  safety  by  introducing  virtual  entities  in  a  scenario; 
unsafe  situations  in  response  to  virtual  entities  are  simply  unacceptable.  Since  displays  can  contain  both  real 
and  virtual  information  at  the  same  time,  operators  should  always  be  aware  which  information  is  real  and 
which  is  virtual.  A  potential  implementation  for  symbols  on  a  display  is  to  give  the  virtual  entities  a  dedicated 
supplementary  tag.  In  the  fighter  embedded  training  system  that  was  developed  at  NLR  in  The  Netherlands, 
this  is  accomplished  by  attaching  a  small  “v”  to  each  virtual  symbol  on  all  displays  where  they  can  appear. 

To  be  successful,  these  issues  must  be  addressed  during  the  early  stages  of  systems  engineering  to  ensure 
proper  human-centered  development  of  UMV  systems  within  a  system-of-systems  architecture.  It  begins  with 
understanding  the  concept-of-operation  in  which  UMVs  will  operate  and  then  identify  mission  system 
requirements.  As  Bruce  Clough  [Reference  26  in  Chapter  7]  correctly  states,  “The  hardest  part  of  making  a 
decision  isn  ’t  deciding,  it ’s  knowing  what  to  decide  with.  ”  What  is  the  situation  and  how  best  can  decision 
aiding  be  applied?  Clough  continues  with  another  lesson  learned,  “Best  autonomy  method  used  is  related  to 
task  to  accomplish,  there  is  no  optimal  method  for  any  task.  ” 

Because  there  is  no  optimal  method,  it  is  critical  that  operators  are  kept  in  the  autonomous  UMV  and  decision 
aid  supervisory  control  loops.  UMVs  are  envisioned  to  operate  in  areas  of  uncertainty,  making  them  subject  to 
automation  “brittleness”.  Automation  brittleness  is  the  concept  that  automated  decision-support  algorithms  are 
typically  fixed  in  code  in  initial  design  phases,  and  therefore  unable  to  resolve  unforeseen  circumstances. 
Higher  levels  of  automation  are  ideal  for  rigid  tasks  that  do  not  require  flexibility  in  decision  making  and  have 
a  low  probability  of  system  failure.  Conversely,  higher  levels  of  automation  are  not  recommended  for 
dynamic  decision  making  environments  like  command  and  control  and  thus  decision  aids  incorporating 
interactive  sensitivity  analysis  are  requisite  because  of  the  risks  and  the  complexity  of  both  the  C2  domain 
system  and  the  inability  of  decision  aids  to  be  perfectly  reliable. 

Following  a  disciplined  Systems  Engineering  approach  that  combines  top-down  requirements  development 
with  a  bottoms-up  rapid  prototyping  capability  should  result  in  a  human-centered  design  that  is  both  optimal 
and  credible.  Using  rapid  prototyping  tools  that  provide  standard  widgets,  display  templates,  and  auto-code 
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generation  allows  the  user  interface  designer  to  produce  concepts  that  can  be  evaluated  early  and  often  by  the 
operator  as  well  as  integrated  directly  with  the  final  mission  control  system. 


1.5  ARTIFICIAL  COGNITION  AND  COOPERATIVE  AUTOMATION 

One  of  the  key  proposed  advances  in  UMV  control  is  the  integration  of  artificial  cognition  in  the  process  of 
vehicle  guidance  and  supervision.  In  particular,  the  idea  of  co-operative  control,  i.e.,  the  co-operation  between 
the  human  operator  and  automation,  is  very  important.  Human-automation  integration  can  be  viewed  from 
two  different  standpoints.  On  the  one  hand,  in  the  near  term,  the  human  has  to  be  considered  as  the  user  of 
automation  technology  not  always  designed  with  the  user  as  the  center  of  the  design.  On  the  other  hand, 
relative  to  the  future,  the  consideration  of  human  performance  in  the  work  processes  suggests  some  unique 
approaches  to  automation  and  decision  systems.  These  approaches  reveal  the  potential  of  human-like 
behaving  and  cooperating  machines  (in  the  sense  of  rational  behaviour)  in  certain  given  task  domains. 
Another  potential  product  is  human-centred  automation,  promising  significant  performance  advances, 
once  introduced  into  a  work  place. 

A  major  emphasis  in  current  conventional  automation  is  the  paradigm  of  supervisory  control.  However, 
with  regard  to  supervisory  control  of  UMVs,  the  operator  can  experience  a  number  of  problems: 

•  Manual  control  of  the  inner  loops  may  not  be  possible  or  desirable  because  of  intolerable  time  delays 
in  the  data  transmission  with  respect  to  the  inner  loop  dynamics  time  constants.  Thus,  the  remote 
operation  relies  heavily  upon  the  availability,  performance  and  integrity  of  some  specific  guidance 
functions. 

•  Insufficient  downlink  bandwidth  and/or  incomplete  sensor  coverage,  with  respect  to  the  task,  can 
cause  what  may  be  called  “keyhole  perspective”  for  the  remote  operator,  potentially  affecting  the 
correctness  or  quality  of  his  or  her  decisions. 

•  The  availability  of  data  link,  i.e.,  the  ability  to  monitor  (via  telemetry)  or  control  (via  telecommand) 
the  vehicle  remotely  may  be  disturbed.  As  a  result,  no  recognition  of  nor  reaction  to  unexpected 
situations  is  possible  any  more  on  the  human  operator’s  side. 

In  essence,  with  the  increasing  complexity  of  automation,  the  human  operator  is  almost  completely  separated 
from  the  underlying  process.  The  long  term  problem  is  loss  of  skills,  i.e.,  erosion  of  competence.  The  human- 
out-of-the-loop  problem  has  other  implications  in  situations  where  operators  almost  fully  rely  upon  the 
automation  -  any  abnormal  situation  will  inevitably  cause  human  overload  and  possibly  erroneous  action. 

One  approach  to  solving  these  problems  is  to  incorporate  an  advanced  automation  concept  called  an  Artificial 
Cognitive  Unit  (ACU)  as  part  of  a  work  system.  Advanced  automation  will  not  displace  the  human  operator  in 
a  work  system,  but  share  the  tasks  in  a  close-partner  work  relationship.  Task  allocation  will  not  be  static, 
but  may  be  adapted  to  the  current  situation’s  needs.  This  includes  the  facilitation  of  redundancy  in  functions 
by  at  least  a  partial  overlap  in  capabilities  with  respect  to  the  task  spectrum.  The  responsibility  of  automation 
(not  necessarily  authority)  will  be  extended  to  the  supervisory  control  level,  i.e.,  automation  will  be  enabled  to 
perform  certain  tasks  under  consideration  of  the  overall  work  system.  Thereby,  automation  brittleness  will  be 
addressed.  Coordination  and  communication  with  such  an  automated  system  will  be  supported  on  all 
performance  levels,  i.e.,  reaching  from  detailed  low  level  information  (reducing  opacity  of  the  machine 
solutions)  up  to  abstract  human-like  information  exchange  on  the  supervisory  level  (addressing  literalism  of 
the  automation).  In  general,  this  approach  to  cognitive  coupling  can  be  a  contributing  factor  to  the  mitigation 
of  the  negative  effects  of  automation  complexity. 
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As  indicated  above,  supervision  and  co-operation  as  accomplishments  of  a  machine  system  require  special 
capabilities.  These  capabilities  were  combined  within  the  notion  of  the  ACU.  Obviously,  the  performance 
feature  of  cognition  is  the  core  element  which  has  to  be  dealt  with  in  order  to  design  such  an  ACU.  From  the 
point  of  view  of  the  discipline  of  cognitive  psychology  human,  i.e.,  natural  cognition  can  be  described  by 
considering: 

•  Perception  and  allocation  of  attention, 

•  Knowledge  representation  and  memory, 

•  Problem  solving,  reasoning  and  decision  making, 

•  Language  comprehension  and  its  generation,  and 

•  Learning  and  the  development  of  expertise. 

The  availability  of  at  least  some  of  these  aspects  of  cognition  is  the  necessary  pre-requisite  to  perform  the 
supervisory  control  task  with  respect  to  the  overall  work  task. 

Within  a  particular  work  system,  the  ACU  represents  all  the  performance  requirements  found  to  be  attributed 
to  the  human  operator  earlier  on,  i.e.,  the  performance  of  decision-making,  problem-solving  and  supervision  in 
order  to  comply  with  the  overall  work  task.  The  implication  of  advances  in  automation  such  as  the  ACU  is  to 
concentrate  on  the  treatment  of  human  and  machine  cognition  as  an  inter-disciplinary  approach  based  upon 
cognitive  psychology  and  artificial  intelligence  as  branch  of  information  technology. 


1.6  CONTROLS  AND  DISPLAYS 

Even  with  rapid  advances  in  computer  processing,  automation  technology,  and  artificial  intelligence  methods, 
there  remains  a  critical  need  for  human  involvement  in  order  for  UMVs  to  successfully  perform  their 
missions.  The  human  provides  unique  strategic  and  innovative  decision-making  capabilities  within  complex, 
dynamic,  and  time  sensitive  situations.  UMV  operator  performance  and,  by  extension,  the  UMV  operator 
control/display  interface,  will  be  even  more  critical  to  achieving  anticipated  new  and  increasingly  complex 
UMV  capabilities  including  close-coupled  operations  with  manned  systems,  UMV  interoperability, 
and  military  strike/combat  operations.  This  chapter  discusses  a  wide  range  of  control  devices.  While  buttons, 
levers,  keyboards  mice,  trackballs,  and  joysticks  are  mentioned,  more  advanced  input  technologies  such  as 
speech  recognition,  touch  pads,  gesture-  and  gaze-based  controls,  receive  the  most  attention.  Physiological 
controls  based  on  electromyographic  and  electroencephalographic  signals  are  also  discussed. 

A  variety  of  display  technologies  are  also  considered.  Visual  displays  include:  head-mounted  and  large  wall- 
mounted,  augmented/mixed  reality,  3d  stereoscopic  and  large  tablet-like  PDAs.  Displays  based  on  other 
senses  take  in  spatial  audio  and  haptic  inputs. 

Given  that  humans  are  to  remain  a  key  component  of  UMV  systems  for  the  foreseeable  future,  it  is  important 
to  recognize  the  unique  challenges  levied  upon  the  operator.  These  challenges  include  the  effects  of  system 
time  delays  (both  fixed  and  variable),  bandwidth  limitations  (which  can  be  intermittent),  datalink 
degradations/dropouts,  and  the  loss  of  the  rich  supply  of  multi-sensory  information  often  afforded  to  onboard 
operators.  With  future  highly  automated  UMV  systems,  issues  also  include  functional  allocation  of  tasks 
between  the  operator  and  the  system,  human  vigilance  decrements,  ‘clumsy  automation’,  limited  system 
flexibility,  mode  awareness,  trust/acceptance  issues,  failure  detection,  and  automation  biases.  However, 
it  is  also  important  to  note  that  the  physical  separation  of  crew  from  vehicle  might  also  offer  some  unique 
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benefits  that  should  be  exploited.  Besides  the  obvious  benefit  to  crew  safety,  it  is  quite  likely  that  available 
bandwidth  and  the  variety  of  available  information  sources  might  be,  in  certain  cases,  far  greater  for  a 
geographically-separated  UMV  crew  versus  an  onboard  operator,  potentially  resulting  in  more  situation 
awareness  rather  then  less.  This,  of  course,  assumes  that  a  well-designed  operator  interface  exists  that  can 
rapidly  filter  and  fuse  this  expanded  information  into  intuitive  displays,  again  underscoring  the  need  to  attend 
to  operator  interface  issues  to  ensure  maximal  system  performance. 

It  is  also  important  to  note  that  as  technology  advances;  the  role  of  the  UMV  operator  must  change  as  well. 
UMV  operator  interfaces  should  not  be  considered  ‘one-size-fits-all’,  but  must  be  tailored  to  match  the 
capabilities  and  limitations  of  the  host  system  and  intended  mission.  Most  current  UMVs  require  that 
operators  have  the  capability  to  manually  control  the  vehicle  and  activate  state  changes  (i.e.,  direct  tele¬ 
operation).  Thus,  operator  interfaces  for  these  vehicles  can  best  leverage  the  numerous  lessons  learned  from 
decades  of  inner-loop  control  design  research,  while  applying  novel  interfaces  to  combat  challenges  that  are 
uniquely  associated  with  UMV  operation. 

With  new,  highly  automated  UMVs,  the  operator’s  role  is  becoming  more  supervisory  in  nature,  overseeing 
the  automated  activation  of  programmed  events  (e.g.,  making  sure  the  appropriate  event  is  activated  at  the 
appropriate  time),  managing  changes  to  the  automated  mission  plan,  and  making  more  strategic-level 
decisions.  These  operator  interfaces  must  take  into  account  issues  associated  with  automation  management, 
including  vigilance  effects,  brittle/clumsy  automation,  sudden  workload  spikes,  etc. 

Continuing  this  trend  beyond  the  current  state-of-the-art,  a  vision  exists  for  a  new  interface  paradigm  for 
controlling  next  generation  UMVs.  This  envisioned  interface  system  involves  multiple  semi-autonomous 
UMVs  being  controlled  by  a  single  supervisor.  These  UMVs  will  have  the  capability  to  make  certain  higher- 
order  decisions,  independent  of  operator  input  and  pre-defined  mission  plans.  This  capability  of  the  UMV 
‘to  decide’  constitutes  a  whole  new  set  of  challenges  for  operators,  as  they  will  be  required  to  rapidly  judge 
the  appropriateness  of  these  decisions  and  assess  their  impact  on  overall  mission  objectives,  priorities,  etc. 
Future  operator  interfaces  will  need  controls  and  displays  tailored  for  multi-UMV  control  and  to  allow  the 
operator  the  capability  to  easily  inspect/override  the  autonomous  UMV  decision-making  logic.  These  interfaces 
will  also  need  to  provide  information  fusing/filtering  algorithms,  intelligent  prioritization/cueing  logic, 
and  possibly  some  form  of  adaptive  task  allocation  in  response  to  rapidly  changing  events  and/or  workload 
levels. 


1.7  HUMAN-AUTOMATION  INTEGRATION 

Many  versions  of  future  concept  of  operations  (CONOPS)  rely  heavily  on  UMVs.  However,  adding  more 
UMVs  and  having  them  perform  more  complex  tasks  will  not  be  realized  without  augmenting  the  current 
structure  of  control.  One  way  to  achieve  this  augmentation  is  through  the  utilization  of  automation. 
Automation,  if  applied  in  a  responsible  and  judicious  manner,  will  enable  the  acquisition  of  capabilities  that 
will  be  required  to  operate  under  near  and  far-term  CONOPS. 

However,  one  of  the  key  questions  is,  exactly  how  will  the  automation  be  applied  in  a  responsible  and 
judicious  manner?  Automation  is  not  a  simple  concept  -  it  involves  different  kinds  of  operator  control, 
function  allocation  between  the  operator  and  the  automation,  various  levels  of  authority  for  the  automation, 
and  the  use  of  intelligent  agents  (single  or  multiples)  within  the  automation.  All  of  these  aspects  have  to  be 
considered,  both  theoretically  and  practically,  if  we  are  to  create  optimal  human-automation  integration. 
Chapter  7  begins  with  a  discussion  of  operator  control  and  finishes  with  UAVs  operating  as  autonomous 
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swarms.  In  the  near  term  operators  will  use  supervisory  control,  but  supervisory  control  is  not  without  its  own 
problems. 

1.7.1  Problems  with  Supervisory  Control  Tasks 

Supervisory  control  of  vehicles  deals  with  automated  vehicle  control  functions  to  a  large  extent.  The  operator, 
who  may  observe  the  controlled  process,  acts  as  a  manager  who  supervises  the  system  and  interacts  with  the 
automated  system  by  performing  corrective  actions.  It  is  known,  however,  that  supervisory  control  systems 
have  certain  limitations  in  performance,  either  on  the  operator’s  side  due  to  human  capacity  limitations  or 
induced  by  deficiencies  of  the  automation,  causing  human  error  intensified  by  the  inability  of  the  automation 
to  perform  on  the  higher  level  of  problem-solving. 

1.7.2  Function  Allocation 

Another  consideration  is  who  should  do  what?  Both  the  operator  and  the  automation  have  the  capability  to 
perform  various  functions.  How  do  we  decide  to  assign  these  functions  to  the  operator  or  the  automation, 
and  once  assigned  how  do  we  integrate  the  human  an  automation  to  work  together  optimally? 

Consider  the  development  of  human  roles  and  automation  from  the  traditional  “left-over  principle”,  through 
human  engineering  optimising  compensatory  principle  with  human  monitoring  (Fitts  lists),  to  contemporary 
complementary  principle  arising  from  human-computer  co-operation/collaboration.  Now  function  allocation 
can  be  dynamic  according  to  external  system  functions,  efficiency  and  system  boundary  conditions. 

1.7.3  Levels  of  Automation 

Once  you  decide  the  allocation  of  the  functions  to  the  automation  or  the  operator,  then  the  question  becomes; 
how  much  authority  do  you  give  to  the  automation  to  act  on  its  own?  Specifically,  how  much  decision  making 
authority  do  you  give  to  the  automation?  These  levels  range  form  none  to  all.  What  are  the  guidelines  to  make 
this  decision? 

The  term  autonomy  has  been  introduced  to  describe  the  bounding  of  functioning  and  decision  authority  of 
advanced  automation  and  intelligent  decision  systems.  Autonomy  can  be  defined  simply  as  the  capability  to 
make  decisions.  Thus,  autonomy  can  be  considered  in  terms  of  the  freedom  to  make  decisions,  considering 
constraints  on  decision-making  (limitations,  boundaries,  rules,  regulations),  decision-making  abilities 
(authority,  responsibility,  competency),  and  the  capability  to  make  different  kinds  of  decisions  (classes, 
functions,  levels). 

For  designing  supervisory  control,  possible  structures  for  the  allocation  of  decision-making  tasks  between 
human  and  computers  are  complex  (up  to  10  levels),  but  some  authors  discuss  four  or  five.  These  have  been 
applied  to  stage  models  of  human  information  processing  functions  (information  acquisition,  analysis, 
decision  selection,  action  implementation).  Ideas  of  levels  of  automation  have  been  proposed  to  represent 
scales  of  delegation  of  tasks  to  automation,  with  implications  for  reliability,  use  and  trust. 

1.7.4  Multi- Agent  Adjustable  Autonomy 

Dynamic  adaptive  and  adjustable  autonomy  is  proposed  for  multi-agent  intelligent  systems  for  distributed 
problem  solving  structures  in  complex  dynamic  environments.  Agents  have  self-direction  and  goals  with 
capability  to  form,  modify  or  dissolve  the  agent  organisation.  Degree  of  autonomy  becomes  linked  to 
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individual  goals.  Focus  moves  to  the  decision  process  for  how  a  goal  is  pursued  free  from  intervention, 
oversight  or  control  by  another  agent.  Autonomy  with  respect  to  goals  is  on  a  variable  scale  (consensus, 
master,  local,  command).  Issues  become  rules  for  transfer  of  control,  communication  protocols,  interaction 
styles,  and  cognitive  strategies  for  reasoning  with  adjustable  autonomy  in  operating  context.  An  example  of 
this  concept  is  illustrated  below  in  the  discussion  of  UAVs  operating  as  a  swarm. 

1.7.5  Levels  of  Automation  within  the  Air  Vehicle 

As  UAV  control  becomes  more  sophisticated,  there  will  be  intelligent  software  both  in  the  operator’s  console 
as  well  as  within  the  UAV  itself.  The  airborne  computing  system  enables  10  levels  of  autonomy  called 
autonomous  control  levels  (ACLs)  within  the  UAV.  One  of  the  interesting  things  about  ACLs  five  and  higher, 
is  that  they  refer  to  how  the  entire  flight  works  together  as  a  group,  with  the  highest  level  being  fully 
Autonomous  Swarms  where  the  vehicles  are  acting  in  concert  with  one  another  to  achieve  a  common  goal. 

So,  what  does  this  have  to  do  with  UAVs?  If  a  flight  of  UAVs  could  act  as  a  swarm,  instead  of  giving  them 
explicit,  detailed  instructions  on  the  location  of  surface-to-air  missile  batteries,  for  example,  they  could  be 
directed  to  just  loiter  about  a  certain  area  of  enemy  territory  and  if  they  come  across  the  missiles  they  could 
destroy  them.  Of  course,  they  would  be  acting  within  the  level  of  responsibility  given  to  them  by  the  human 
operator.  Creating  digital  pheromones  for  UAVs  is  one  way  they  could  communicate.  These  types  of 
pheromones  are  not  based  on  chemicals,  but  rather  on  the  strength  of  electrical  fields.  In  a  computer-based 
(constructive)  simulation,  a  UAV  swarm  using  digital  pheromones  significantly  outperformed  the  non-swarm 
case. 

1.8  PUTTING  IT  ALL  TOGETHER 

Although  some  progress  has  been  made  -  there  are  UMVs  operating  in  various  areas  of  the  world  today  - 
no  integrated  theory  of  human-automation  integration  has  surfaced  as  of  this  writing.  Perhaps  this  should  not 
be  a  surprise.  The  integration  of  humans  and  automation  is  what  is  called  a  “wicked”  problem,  one  not 
answered  by  simple  solutions.  However,  the  fact  that  UMVs  are  operational  gives  us  hope  that  the  problem  is 
not  intractable.  In  addition,  ideas  expressed  in  the  following  chapters  offer  a  great  potential  for  solving  this 
“wicked”  problem. 
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Chapter  2  -  MILITARY  RELEVANCE 

Chapter  Lead:  R.  Taylor 

Contributors:  G.  Carver,  P.  Kennett,  P.  Stensson,  R.  Taylor 

2.1  PART  I  -  MILITARY  RELEVANCE  OF  HUMAN  FACTORS  OF  UMV 
SYSTEMS 

2.1.1  Introduction 

“Now  it  is  clear  the  military  does  not  have  enough  unmanned  vehicles.  We  are  entering  an  era  in 
which  unmanned  vehicles  of  all  kind  will  take  on  greater  importance  -  in  space,  on  land,  in  the 
air  and  at  sea.  ” 

President  George  W  Bush,  address  to  the  Citadel,  December  2001. 

Unmanned  Military  Vehicles  (UMVs)  are  enablers  of  military  capability  with  clear  endorsement  at  the  highest 
level.  Many  NATO  Nations  have  active  programmes  to  develop  and  integrate  UMV  systems  into  the  front  line 
military  force  mix.  UMVs  are  most  commonly  characterised  as  dealing  well  with  3D  tasks  -  dull,  dirty  and 
dangerous.  They  are  used  extensively  in  intelligence,  surveillance  and  reconnaissance  (ISR)  roles,  affording 
persistence  in  the  provision  of  critical  information,  without  risking  lives.  Increasingly,  they  are  being 
considered  for  combat  and  support  roles.  Modern  warfare  needs  military  capability  to  respond  to  the  threat  of 
conventional  hostile  force  and  to  the  challenges  of  asymmetric  conflicts,  where  political  and  military 
success  relies  on  effects-based  targeting  and  operations.  In  the  age  of  Network  Centric  Warfare  (NCW), 
ISR  information  supplied  by  UMV  systems  can  be  a  key  combat  multiplier  in  the  hands  of  a  commander  [1]. 
Automation  technology  and  computer-based  information  processing  are  increasingly  important  for  balancing 
affordability,  capability  and  achievability  with  increasing  pressures  on  scarce,  skilled  human  resources. 
Important  questions  remain  about  what  realistic  effects  can  be  expected  to  be  achieved  by  UMVs  in  the 
uncertain,  ambiguous  and  non-linear  battle-space  of  the  future,  including  how  international  law  will  interpret 
robotic  warfare  in  the  future  [2,3].  However,  the  main  consideration  of  this  report  is  not  so  much  the  military 
relevance  of  UMVs,  since  this  seems  mostly  self  evident.  Rather,  the  key  issue  is  to  establish  why  human 
factors  (HF)  are  important  military  relevant  issues  with  “unmanned”  technologies.  Since  UMV  technologies 
are  expected  to  actually  reduce  human  involvement  in  some  tasks,  it  is  not  self  evident  why  HF  issues  should 
warrant  raised  attention.  Thus,  the  reasons  for  raising  HF  concerns  are  in  need  of  review.  In  this  report,  when 
referencing  UMVs,  the  term  “uninhabited”  will  be  substituted  for  “unmanned”  where  appropriate, 
in  recognition  of  the  role  of  both  women  and  men  equally  in  serving  our  armed  forces. 

2. 1.1.1  References 

[1]  Curran,  M.  (2005).  UAVs:  A  Critical  Multiplier  for  Current  and  Future  Forces.  RUSI  Defence  Systems, 
Summer  2005.  pp.  64-66.  London,  Royal  United  Services  Institute. 

[2]  Burridge,  B.  (2005).  Post-modern  warfighting  with  unmanned  vehicle  systems:  Esoteric  chimera  or 
essential  capability.  RUSI  Journal,  October  2005,  150  (5).  London,  Royal  United  Services  Institute. 

[3]  Kennett,  P.D.  (2005).  Autonomous  Killing  Machines  -  The  Technical,  Legal  and  Moral  Implications. 
Defence  Research  Paper,  Advanced  Command  and  Staff  Course  No  8,  September  04  -  July  05,  Joint 
Services  Command  and  Staff  College,  March  2005. 
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2.1.2  Human  Factors 

2. 1.2.1  User  Requirements 

UMVs  are  valued  variously  as  force-multipliers,  as  augmenters  of  the  force,  and  as  adding  a  new  component 
to  the  military  force  mix  -  but  ultimately,  UMVs  are  tools  for  human  use.  Human  effectiveness  is  the  key  to 
all  military  capability.  UMVs  are  enablers  of  human  effectiveness  and  military  capability.  Human 
involvement  in  UMV  systems  is  of  paramount  importance.  HF  issues  need  to  be  in  the  sharpest  focus  to 
mitigate  the  unacceptable  risks  of  de-humanisation  of  decision-making  in  warfare. 

Air  Chief  Marshall  Sir  Brian  Burridge  believes  that  in  order  to  appreciate  the  capability  of  UAVs,  we  need  to 
appreciate  their  limitations  and  benefits,  but  that  understanding  of  the  human  dimension  is  the  most  important 
of  all  -  knowing  how  to  use  them,  task  them  and  to  integrate  them  [1].  Use  of  UMVs  is  generally  justified  on 
grounds  of  capability,  affordability  and  safety.  UMVs  can  make  certain  tasks  safer  by  reducing  human 
involvement  and  risk  to  life,  allowing  the  possibility  of  human  resources  being  re-deployed  more  efficiently 
and  effectively.  This  produces  complex  changes  in  the  balance  and  priority  of  HF  issues  for  UMV  systems. 
Paradoxically,  for  many  aspects  of  UMV  system  engineering  and  operation,  the  proper  consideration  of  HF 
has  even  greater  military  relevance.  Human  involvement  remains  essential  throughout  the  UMV  system  life 
cycle,  including  UMV  operations.  As  a  discipline,  HF  provides  the  tools  for  understanding  and  ensuring  the 
correct  human  involvement  in  the  UMV  system  life  cycle.  Obviously,  UMV  habitability  is  not  a  concern. 
However,  vehicle  maintenance  is  still  needed.  Vehicle  control  and  safety  becomes  a  complex  issue,  especially 
when  mixing  UMVs  with  manned  vehicles  and  “dismounted”  forces.  Increasing  levels  of  UMV  autonomy  are 
expected  to  reduce  the  need  for  human  intervention  in  operations.  However,  UMVs  are  not  a  substitute  for 
human  involvement  in  the  battle-space.  Crucially,  human  control  of  UMVs  is  axiomatic  for  military  relevance 
(for  a  detailed  argument  for  the  axiomatic  requirement  for  human  control,  see  Chapter  2,  Part  II). 
Consideration  of  the  technological  viability  of  autonomous  systems,  and  the  legal  constraints,  suggests  that  a 
“human-in-the-loop”  system  will  be  the  most  valuable  and  therefore  the  most  likely  mode  of  operation  to 
provide  the  required  supervision  and  discrimination  [2]. 

2. 1.2. 1.1  Battle-Space  Connectivity 

In  warfare,  the  problems  and  outcomes  are  complex,  dynamic,  uncertain  and  risky,  and  the  application  of 
critical  judgement  and  decision  making  is  crucial  to  successful  conflict  resolution.  Context  sensitivity  is 
important  for  assessing  the  quality  of  military  decision  making  [3].  Humans  encode  context  naturally  and 
handle  decision  making  adaptively  with  incomplete,  partial  and  uncertain  information.  This  provides  decision 
making  capability  not  easily  matched  by  artificial  intelligence  (AI)  in  computers.  However,  to  exercise  good 
military  judgement,  humans  need  to  feel  the  texture  and  “granularity  of  the  battle-space”  [1].  UMV  operators 
removed  from  the  immediate  context  of  use,  risk  losing  “emotional  connectivity  of  the  battle-space”,  operator 
context  sensitivity  and  system  adaptiveness.  For  autonomous  UMV  operations,  the  detailed  level  of  operator 
supervision  required  is  likely  to  be  dependent  on  the  individual  mission  context  and  the  Rules  of  Engagement 
(RoE).  This  can  be  difficult  or  impossible  to  anticipate  fully  in  advance.  As  a  minimum,  the  operator  needs  to 
be  able  to  discriminate  between  what  is  a  valid  military  target  and  what  is  not. 

2. 1.2. 1.2  Human  in  Control 

Technological  limitations,  legal  and  moral  constraints,  and  most  effective  human  involvement,  suggest  that 
some  form  of  human-in-the-loop  control  always  will  be  required.  Currently,  with  manned  vehicles,  the  human 
operator  provides  the  flexibility  to  adapt  to  constraints  on  functioning  arising  from  system  design,  creates 
on-line  tolerance  of  variability  and  uncertainty  in  the  external  environment,  and  offers  adaptation  to  changing 
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dynamic  mission  context.  The  requirements  for  human-in-the-loop  control  of  UMV  operations,  either  remote 
or  reach-back,  can  be  considered  as  occurring  at  a  number  of  levels  depending  on  the  level  of  automation, 
e.g.,  tasks,  functions,  tactical  and  strategic  mission  goals.  Classes  of  control  can  be  characterised  as  either 
manual,  semi-automatic,  and  fully  autonomous,  with  and  without  human  supervisory  control.  Generally, 
automation  best  serves  human  purposes  by  enabling  higher  levels  of  human  control,  i.e.,  automate  routine  3D 
tasks  and  support  human  supervisory  control  at  tactical  and  strategic  levels.  The  challenge  is  to  determine  the 
precise  level  of  supervision  required,  and  to  identify  the  detailed  user  requirements  and  HF  engineering 
solutions,  for  efficient  and  effective  supervisory  control. 

In  highly  autonomous  operations,  communications  permitting,  humans  can  retain  high  level  supervisory 
control  through  setting  and  monitoring  of  tasks  and  goals,  and  through  authorisation  of  safety  critical  actions 
and  use  of  lethal  force.  However,  experience  of  automation  supervision  elsewhere,  in  particular  in  the  process 
control  industry  and  with  flight  deck  automation  has  shown  that  reliable  and  robust  human  supervisory  control 
is  inherently  difficult  to  achieve.  Dependence  on  human  supervisory  control  is  risky  for  safety  critical  events 
and  tasks.  Limitations  on  cognition  (perception,  learning,  memory  and  reasoning)  mean  that  it  is  inherently 
difficult  for  humans  to  perform  supervisory  control  in  a  consistent  and  reliable  manner,  particularly  during 
sustained  operations  requiring  vigilance  and  unpredictably  intermittent  high  levels  of  attentional  engagement. 
Ultimately,  there  is  a  risk  that  the  over-use  of  automation  may  reduce  human  authority,  responsibility  and 
competency.  Crucially,  over-use  of  automation  risks  de-skilling  the  user  in  the  important  cognitive  domain, 
reducing  the  essential  human  capability  for  exercising  critical  judgement  and  decision  making  in  the 
appropriate  use  of  lethal  force.  Finally,  over-use  of  automation  implies  the  risk  that  the  human  supervisor  is 
placed  ‘out-of-the  loop’  so  that  he  lacks  actual  process  state  knowledge  (so-called  peripherisation  effect). 

Supervisory  control  requires  robust  and  reliable  communications  with  the  battle-space.  The  realities  of 
military  communications  present  a  real  dilemma  for  the  supervisory  control  paradigm.  In  practice, 
communications  technology  limitations  (e.g.,  line-of-sight  and  bandwidth  restrictions,  information  quality, 
latency)  and  communications  breakdown  (e.g.,  hostile  interference,  electronic  countermeasures)  can  limit 
feedback  on  mission  performance  and  prevent  real-time  mission  intervention  during  remote  control 
operations.  This  may  necessitate  detailed  mission  planning,  including  contingencies  for  restricted  autonomous 
operations  when  human  supervision  and  authorisation  is  denied. 

2. 1.2. 1.3  Authority  and  Responsibility 

Human  involvement  is  required  in  military  operations  to  direct  and  plan  the  use  of  military  capability,  and  to 
ensure  lawfully  correct  use  of  lethal  force.  This  is  achieved  through  the  application  of  human  command 
authority,  responsibility  and  accountability,  and  competency.  With  autonomous  UMVs,  some  of  that 
responsibility  is  delegated  to  increasingly  competent  computer  controlled  machines,  but  the  authority  and 
accountability  for  the  delegation  ultimately  remains  with  humans.  Ensuring  the  correct  human  involvement  in 
UMV  operations  provides  issues  for  Command  and  Control  (C2),  concept  of  operations  (CONOPS),  ROE  and 
for  the  specific  information  display  and  control  requirements  in  the  context  of  use,  i.e.,  ISR,  combat, 
or  support  roles.  For  military  relevance,  UMV  autonomy  concepts  must  be  integrated  with  the  C2 
requirements  of  both  national  and  international  C2  infrastructures  (joint  and  coalition  operations).  C2  is  rooted 
in  human  authority,  responsibility  and  accountability,  will,  leadership  and  competency  in  judgement  and 
decision  making  [4,5].  The  potential  for  fully  autonomous  UMV  operations  presents  significant  challenges  for 
concepts  and  principles  of  military  C2.  UMV  control  requirements  need  to  be  integrated  with  C2  frameworks 
and  architectures  (information  flows,  decision  nodes,  dynamic  interactions),  chains  of  command  and 
CONOPS.  This  is  to  ensure  leadership  and  the  correct  delegation  of  human  authority,  responsibility  and 
accountability,  and  the  necessary  dynamic  human  interactions,  with  appropriate  levels  of  trust. 
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2. 1.2. 1.4  Comprehension  of  Meaning 

UMVs  are  used  extensively  to  gather  information  in  ISR  roles  for  human  interpretation.  ISR  information  is 
inherently  incomplete  and  uncertain.  Fundamentally,  computer-based  information  processing  systems  are 
limited  in  that  they  can  not  comprehend  the  meaning  of  information  in  human  cognitive  terms,  e.g.,  apply 
knowledge,  understand,  feel  truth,  appreciate  implications,  judge  consequences.  Critical  military  judgement  is 
needed  to  interpret  the  meaning  of  ISR  information.  Crucially,  UMVs  can  not  appreciate  the  effects  of  use  of 
lethal  force.  An  “emotional  connectivity”  is  needed  to  appreciate  the  “moral  value  of  killing  and  the  value  of 
human  life”  [2].  Critical  military  judgement  is  needed  for  decisions  on  use  of  lethal  force.  Failure  to  ensure 
proper  human  involvement  risks  rendering  UMVs  as  unusable  tools  for  military  purposes. 


2. 1.2.2  Integration  of  Human  and  Automation  Requirements 

As  a  speciality,  HF  is  traditionally  concerned  with  the  study  of  the  man-machine  interface.  This  also  includes 
consideration  of  the  equipment,  the  physical  environment,  the  tasks  and  the  individuals  who  do  the  work. 
Humans  are  involved  throughout  the  UMV  life  cycle,  from  conceptualisation,  specification,  design  and 
development,  through  command,  control,  operation  and  maintenance,  to  decommissioning.  The  term  Human 
Systems  Integration  (HSI)  is  increasingly  used  in  NATO  nations  to  cover  the  broad  scope  of  human 
considerations  needed  from  a  human-centred  approach  to  systems  engineering  (or  in  a  system-of-systems 
approach).  The  following  definition  of  HSI  has  been  agreed  by  NATO  NSA  Aircrew  Integration  Panel  for 
addition  to  AAP-45  (NATO  Glossary  of  Aircraft  -  Aircrew  Integration  (AI)  Specialist  Terminology  and 
Abbreviations):  “The  technical  process  of  integrating  the  interdependent  elements  of  Human  Factors 
Engineering,  Manpower,  Personnel,  Training,  System  Safety,  Health  Hazards,  Survivability,  and  Habitability 
into  the  system  acquisition  process  to  ensure  the  safe  and  effective  operability  and  supportability  with 
minimised  Life  Cycle  Cost  (LCC)”.  UMVs  change  the  challenges  of  system  safety,  health  hazards, 
survivability  and  habitability,  reducing  risks  compared  with  manned  vehicles,  particularly  for  remote  “reach- 
back”  operations.  Otherwise  the  HSI  domains  of  HF  Engineering,  Manpower,  Personnel,  Training,  remain  as 
ever  highly  relevant  for  the  UMV  system  life  cycle.  Notwithstanding,  achieving  the  correct  human- 
automation  integration  is  a  key  HSI  challenge  for  UMVs,  with  significant  implications  for  HF  Engineering, 
Manpower,  Personnel  and  Training. 

2. 1.2.2. 1  Manpower 

Generally,  UMVs  are  expected  to  augment  the  force  and  to  create  potential  savings  in  human  resources, 
manning  levels  and  training.  However,  Air  Chief  Marshall  Sir  Brian  Burridge,  Commander  in  Chief,  Strike, 
of  the  Royal  Air  Force  (RAF)  describes  how  the  current  manpower  burden  of  remotely  piloted  operations  is 
significant,  and  should  not  be  underestimated  [1].  The  Air  Chief  Marshall  reports  that  as  part  of  the  Predator 
Task  Force  at  Nellis  Air  Force  Base,  the  RAF  mans  a  single  Predator  A  Orbit  in  support  of  the  coalition 
intelligence,  surveillance  and  reconnaissance  effort.  This  RAF  unit,  115  Flight,  totals  44  RAF  personnel. 
He  explains  that  a  Predator  A  can  orbit  for  20  hours  and  requires  2  crew  who  operate  for  8  hours  each, 
totalling  6  crew  for  a  single  Predator.  In  addition,  the  operation  involves  analysts,  data  link  managers, 
engineers  in  the  deployed  location,  and  the  crews  required  to  launch  and  to  recover  the  UAV  in  theatre. 
This  corresponds  to  a  considerable  manpower  intensive  effort,  in  stark  contrast  with  the  current  aspiration  of 
UAVs  to  reduce  the  manpower  burden.  In  the  future,  UMVs  may  be  expected  to  operate  with  increased  levels 
of  autonomy,  with  concomitant  reductions  in  human  involvement  in  platform  control.  Estimates  of  savings 
require  comparisons  with  manned  aircraft  operations  providing  the  same  level  of  persistence  -  but  on  the 
evidence  from  Predator  A  operations,  we  should  be  careful  not  to  underestimate  the  human  resources 
requirements  of  UMV  operations. 
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Esker  [6]  describes  how  the  Global  Hawk  High  Altitude  Endurance  UAV  system,  with  relatively  higher  levels 
of  autonomous  functioning  compared  with  Predator,  has  ground  control  facilities  comprising  two  elements, 
the  Launch  and  Recovery  Element  (LRE)  and  the  Mission  Control  Element  (MCE).  The  LRE  accommodates 
two  persons  and  is  responsible  for  pre-flight  and  post-flight  ground  operations  and  the  takeoff  and  landing 
phases  of  flight.  The  MCE  accommodates  four  persons  and  is  responsible  for  the  mission  portion  of  the  flight, 
when  the  vehicle  is  at  cruise  altitude. 

Currently,  it  is  a  priority  in  many  NATO  Nations  UAV  research  programmes  to  reduce  the  manpower  burden 
by  reducing  the  ratio  of  operators  to  vehicles  for  flight  and  mission  control.  A  common  aim  is  to  increase 
operator  effectiveness  by  enabling  a  single  operator  to  control  multiple  UAVs  simultaneously  (typically  up  to 
four)  by  introducing  increased  levels  of  automation,  operator  decision  aiding  and  advanced  human-computer 
interfaces  (HCI). 

2. 1.2. 2. 2  Personnel  and  Training 

Air  Chief  Marshall  Sir  Brian  Burridge  considers  that  softer  human  issues  associated  with  operator  selection 
and  training  need  to  be  addressed  urgently,  ahead  of  some  of  the  technological  issues  [1].  Predator  is  remotely 
operated  by  a  pilot  and  a  sensor  operator.  Other  UAVs  use  a  computer  operator.  The  Air  Chief  Marshall 
expresses  concern  that  without  proper  training,  operators  “could  be  faced  with  the  very  real  possibility  of 
unwrapping  one  of  these  systems  for  the  first  time  on  operations”.  Integration  and  interaction  with  civilian 
airspace  constraints  is  a  key  training  issue.  He  emphasises  the  importance  of  previous  military  experience 
gained  in  operations.  The  experience  of  operating  manned  platforms  enables  them  to  interact  with  other  units 
and  to  operate  safely  within  airspace.  He  notes  that  they  understand  the  needs  of  other  units  through  this 
shared  connection.  Solutions  may  need  to  be  found  in  the  selection  of  personnel  with  appropriate  operational 
experience,  and  in  creating  an  appropriate  work  context  for  proper  operator  task  engagement  through  a 
combination  of  HF  engineering,  HCI  and  training  in  RoE  and  effects  appreciation. 

In  the  future,  the  possibility  of  increasingly  autonomous  UMVs  can  be  expected  to  place  greater  cognitive 
demands  on  the  operator,  with  little  or  no  manual  control  required.  Basic  military  skills  and  knowledge  will 
continue  to  be  important,  such  as  airmanship  and  seamanship.  The  role  of  psychomotor  abilities  will  become 
diminished.  Performance  of  tasks  that  are  likely  to  be  required  include: 

•  Managing  and  controlling  multiple  UMV  missions; 

•  Co-ordination  and  de-confliction  of  multiple  UMV  assets; 

•  Interpreting  and  integrating  command  strategic  intent,  RoE  and  mission  control  requirements; 

•  Recognizing  and  dealing  with  degraded  system  functionality; 

•  Regaining  SA  after  loss  of  UMV  data  links; 

•  Interpreting  displays  containing  multiple  UMV  perspectives; 

•  Shift  of  system  control  to  other  team  members  or  control  stations;  and 

•  Team- working  and  interpersonal  interaction. 

The  emphasis  will  be  on  the  UMV  operator  as  a  mission  manager,  on  multi-task  management  and 
performance,  on  judgement  and  decision-making  skills,  and  on  the  cognitive  ability  to  integrate  and  interpret 
dynamic,  complex  data,  in  order  to  make  rapid  and  effective  decisions.  For  a  detailed  discussion  of 
manpower,  skills  and  training  requirements,  see  Chapter  4. 
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2. 1.2. 2. 3  HF  Engineering 

From  a  system-of-systems  point  of  view,  the  term  “unmanned”  is  potentially  misleading.  It  is  most  certainly 
inappropriate  from  an  HF  engineering  perspective.  It  suggests  an  absence,  or  a  reduction,  in  human 
involvement,  and  consequently  a  lack  of,  or  a  lessening,  in  human  system  issues.  This  is  particularly 
unfortunate  since  the  opposite  is  probably  nearer  the  truth.  UMVs  remove  humans  from  the  vehicle 
(or  platform)  and  the  hazardous  operating  environment.  However,  UMVs  do  not  remove  humans  from  the 
system  of  use.  At  the  present  time,  with  human-in-the-loop  control,  advances  in  autonomous  vehicle 
technologies  are  worthless  without  an  effective  and  efficient  operator  remote  control/display  interface  [7]. 

Generally,  separating  operators  from  the  context  of  use  risks  disconnection  from  the  battle-space,  reduced  SA, 
and  creates  difficulty  in  decision  making  and  in  maintaining  the  level  of  control  and  feedback  on  the  effects  of 
use.  Rather  than  reducing  the  human  system  issues,  increasing  remoteness  may  risk  reducing  the  operator’s 
capability  to  provide  effective  task  engagement,  situation  appreciation  and  timely  interventions.  Increased 
levels  of  autonomy  may  reduce  some  of  the  human-in-the-loop  workload,  but  autonomy  risks  the  effects  of 
disconnection  identified  above.  Research  is  needed  both  on  AI  techniques  for  autonomy  and  on  HF  of 
supervisory  control.  For  a  detailed  discussion  of  relevant  AI  techniques,  see  Chapter  5.  The  risk  of 
disconnection  raises  the  importance  of  HF  engineering  for  enabling  supervisory  control  and  for  exploiting  the 
potential  mitigations  afforded  by  advanced  HCI  design,  augmented  cognition  technologies,  SA  tools  and 
operator  decision  support  aids.  HCI  style  guide  information  is  available  for  interoperability  between  UAV 
Control  Stations  (UCS)  in  NATO  STANAG  4586  [8].  HCI  is  a  rapidly  advancing  field  and  research  is  needed 
to  provide  properly  validated  advanced  HCI  solutions  for  future  improved  UCS.  For  a  detailed  discussion  of 
advanced  UMV  operator  interfaces  control/display  requirements,  see  Chapter  6. 

It  has  been  suggested  that  UMVs  may  shift  the  balance  of  responsibility  and  accountability  for  UMV 
behaviours  and  effects  from  users’  decisions  during  systems  operation  towards  engineers’  decisions  during 
system  design.  The  development  of  this  argument  could  depend  on  technological  developments  affecting  the 
future  possibility  of  machines  that  never  make  mistakes,  the  levels  of  automation  employed,  and  the  methods 
of  supervisory  control  of  operations  and  effects.  International  humanitarian  law  on  military  use  of  lethal  force 
in  conflict  seems  likely  to  keep  responsibility  and  accountability  firmly  with  the  user/commander.  So,  with 
increasing  levels  of  autonomy,  the  need  for  system  transparency  and  SA  could  grow.  User  involvement  in 
systems  requirement  specification  will  become  increasingly  important  to  ensure  that  critical  military 
judgement  can  be  properly  exercised  in  the  context  of  use.  This  could  necessitate  real  time  user/commander 
control  of  the  level  of  automation  (i.e.,  adjustable,  variable  levels  of  automation),  in  addition  to  supervisory 
control  of  the  specific  UMV  operations  and  effects. 

2. 1.2. 2. 4  Human-Computer  Decision  Partnership 

Reising  [9]  describes  how  in  the  future,  rather  than  coping  with  unreliable  human  supervisory  control, 
or  simply  removing  the  operator  from  the  control  loop  entirely,  the  paradigm  for  operator  control  will  need  to 
progress  to  one  based  on  human-computer  co-operation,  as  implemented  in  advanced  pilot  assistance  systems. 
Reising  predicts  that  future  UMVs  will  contain  associate  systems  that  will  enable  the  UMV  operator  and  the 
associate  to  form  a  team  of  two  crewmembers  -  one  human  and  one  electronic.  Onken  [10]  calls  this 
partnership  cognitive  co-operation.  Ensuring  the  success  of  this  necessary  partnership  presents  significant 
challenges  for  HF  of  UMV  systems.  Research  has  shown  how  real-time  HF  engineering  of  variable  levels  of 
automation  or  adjustable  levels  of  autonomy  are  important  for  controlling  multiple  autonomous  UMVs, 
and  provide  the  key  to  developing  an  adaptive  human-computer  decision  partnership  [11,12,13].  Alternative 
theoretical  frameworks  for  UMV  systems  are  discussed  in  Chapter  3.  Levels  of  automation  are  discussed  in 
detail  in  Chapter  7. 
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Ideally,  a  flexible  approach  is  needed  that  allows  a  variable  level  of  human  intervention  and  autonomy, 
with  the  need  for  “drill-down”  judged  in  real-time.  For  efficient  and  effective  mission  supervision  and 
discrimination,  the  operator/supervisor  needs  to  be  able  to  bring  added  value  to  the  understanding  of  the 
situation  [14].  To  add  value,  he/she  will  need  to  be  able  to  use  knowledge  (e.g.,  RoE,  situation  awareness, 
tactics)  and  to  take  into  account  additional  contextual  decision  information  not  available  to  the  UMV 
information  processing  system.  Otherwise,  the  level  of  supervision  may  be  uncritical  and  lack  any  real 
operator  decision  input,  with  the  resultant  legal  implications.  One  example  to  avoid  would  be  authorising 
target  prosecution  for  autonomous  UMVs  based  only  on  pre-set  automatic  target  recognition  (ATR)  criteria 
without  independent  operator  verification  of  the  target  context  RoE.  To  mitigate  this,  the  C2  system  and  UCS 
need  to  provide  a  rich  operating  picture  for  mission  assessment  and  appropriate  mission  performance 
critiquing  tools. 
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2.1.3  Operational  Benefits  of  UMVs 

Interoperability  within  the  force  mix  is  a  key  challenge  for  UMVs  and  HF.  Commodore  Lambert,  UK  Director 
of  Equipment  Capability  (Underwater  Battle-space)  has  characterised  the  battle-space  of  today  as 
encompassing  air,  land  and  sea  domains,  requiring  sensor  coverage  and  connectivity  across  all  boundaries  [1]. 
In  this  battle-space,  Commodore  Lambert  has  identified  several  operational  advantages  in  military  missions 
derived  from  employing  UMVs  in  airborne,  ground  and  underwater  systems: 

•  Minimise  or  eliminate  risk  to  personnel  and  expensive  platforms  through  autonomy. 

•  Access,  under  all-conditions,  to  denied  or  unsafe  areas  of  operations. 

•  Force  multiplication  through  the  ability  to  operate  independently  for  extended  periods,  enabling 
manned  platforms  to  extend  their  reach  and  focus  on  more  complex  tasks. 

•  Automated  reconnaissance,  surveillance,  target  acquisition,  target  designation,  tactical  oceanography, 
battle  damage  assessment,  etc.,  through  the  use  of  miniaturised,  low  energy  sensors/payloads. 

•  Secure  network  enabled  warfare  via  data  relay  and  connectivity  at  extended  ranges  between  multiple 
vehicles. 

•  Mission  flexibility  through  their  ability  to  be  deployed  from  a  variety  of  host  platforms. 

Of  the  three  main  classes  of  UMVs  -  airborne,  ground  and  underwater  systems  -  uninhabited  air  vehicles 
(UAVs)  and  to  a  lesser  extent,  uninhabited  underwater  vehicles  (UUVs)  have  attracted  the  wider  operational 
interest.  UAVs  have  probably  resulted  in  the  most  significant  technical  activity  from  a  human  factors 
perspective.  The  following  sections  reflect  this  balance  of  military  interest,  HF  issues  and  research  activity. 


2. 1.3.1  Uninhabited  Ground  Vehicles 

Uninhabited  ground  vehicles  (UGVs)  were  first  used  by  the  German  Army  in  World  War  II  for  tasks  such  as 
mine  field  breaching  (e.g.,  Goliath,  Borgward  IV).  Early  UGVs  have  been  particularly  successful  in  support  of 
space  operations,  such  as  the  Lunar  and  Mars  rover  vehicles.  The  first  such  vehicle  was  the  Russian  Lunakhod 
1  which  was  tele-operated  on  the  moon  in  the  1970s  for  11  days  over  a  distance  of  10  km.  Currently, 
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the  majority  of  UGVs  are  tele-operated  line-of-sight  systems,  with  research  directed  at  the  development  of 
autonomous  unmanned/robotic  technology.  Strong  [2]  summarises  the  potential  use  of  UGVs  as  including  the 
following  roles: 

•  Mine  detection  and  neutralisation; 

•  Explosive  ordnance  clearance  and  disposal; 

•  Reconnaissance,  surveillance  and  target  acquisition; 

•  Operations  in  contaminated  areas  and  contamination  assessment  (e.g.,  detection  and  analysis  of 
chemical,  nuclear  and  biological  assets); 

•  Urban  warfare  operations  in  confined  spaces; 

•  Fire  fighting; 

•  Logistics  (e.g.,  delivery  to  the  battlefield  of  munitions,  fuel,  parts,  food  and  water); 

•  Casualty  recovery; 

•  Security  and  patrol; 

•  Deployment  of  weapons  or  obscurants; 

•  Deployment  as  a  mobile  communications  link; 

•  Deployment  as  a  mobile  power  supply;  and 

•  Deployment  as  a  decoy  target. 

Current  UGVs  have  important  military  roles  for  reconnaissance  and  surveillance  in  support  of  urban 
operations,  particularly  for  working  in  confined,  restricted  and  dangerous  environments,  such  as  fire-fighting 
(e.g.,  Carlos),  breaching  (e.g.,  Caterpillar  D7),  searching  underground  in  caves,  culverts,  drains  and  man  holes 
and  searching  collapsed  buildings,  i.e.,  areas  not  subject  to  aerial  observation  (e.g.  PackBots,  Man  Portable 
Robotic  System).  Generally,  for  military  purposes,  UGVs  experience  major  challenges  to  mobility  and 
manoeuvrability  due  to  sensing  and  avoidance  difficulties  with  unexpected  ground  obstacles  and  crevices,  and 
due  to  terrains  (rugged  going,  gradients,  instability,  terrain,  grip  and  ground  clearance;  building  sites  and 
interiors;  smooth  surfaces;  cluttered  environments;  low  tide,  streams,  puddles,  mud).  Research  has  shown  that 
failure  rates  in  urban  terrain  are  relatively  high  with  a  mean-time  between  failures  on  average  of  between 
6-20  hours.  Because  of  the  relatively  restricted  nature  of  UGV  tasks,  they  are  relatively  unsophisticated  in 
terms  of  requirements  for  levels  of  automation  and  autonomous  functioning.  UGVs  are  mostly  operated  by 
remote  control,  requiring  robust,  compact  and  portable  operator  control  stations,  with  relatively  simple  control 
and  display  interfaces. 

Remotely  controlled  UGVs  have  a  significant  current  role  as  tools  for  detecting  hazardous  and  dangerous 
materials  (chemical,  biological,  radiation,  nuclear  -  CBRN),  and  in  particular  for  counter-mine,  Explosive 
Ordnance  Devices  (EOD)  and  Improvised  Explosive  Devices  (IED)  tasks,  e.g.,  RONS  -  Remote  Ordnance 
Neutralisation  System;  ARTS-A11  purpose  Robotic  Transportation  System;  M60  Panther  (Common  Robotic 
System),  a  tele-operated  turret-less  Ml  tank  equipped  with  rollers  for  mine  clearance;  Mini  Flail  for  anti¬ 
personnel  mines  and  booby  traps;  Wheelbarrow  tele-operated  tracked  robotic  bomb  disposal  vehicle;  Cyclops, 
and  Buckeye  Miniature  Remotely  Operated  Vehicle  (MROV)  for  work  in  urban  areas  and  confined  spaces, 
such  as  aircraft,  buses  and  trains;  Talon,  a  remotely  operated  vehicle-based  telescopic  manipulator,  acts  as 
both  a  reconnaissance  vehicle  and  a  weapon  and  camera  platform.  The  current  version  of  Talon  is  a  semi- 
autonomous  unmanned  vehicle  capable  of  firing  rifles,  machine  guns,  grenade  launchers  and  rockets. 
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Control  of  UGVs  for  EOD/IED  operations  involves  the  following  HF  challenges: 

•  Systems  operations  difficulties  -  power  supply  management,  ambient  lighting  problems,  visual  issues, 
control  feedback. 

•  System  performance  issues  -  lag  and  gain  in  control  response  critical  for  EOD/IED  work,  stopping 
distance,  proportional  gain  controls,  mode  types,  simultaneous  control  of  multiple  parameters. 

•  Operating  from  moving  platforms  -  motion  sickness,  orientation. 

In  addition,  the  EOD/IED  operational  environment  involves  complex  problem  solving  and  high  levels  of 
operator  situational  awareness.  This  requires  high  levels  of  operator  and  team  skills  and  experience.  Training 
of  EOD/IED  operators  is  an  important  area  for  HF  work.  This  summary  of  EOD/IED  HF  requirements  is 
based  on  the  UGV  briefing  by  Paul  Burns,  SOE  Academy  and  Nicki  Heath,  Symbiotics,  provided  to 
HFM-018  at  the  meeting  in  Bath,  UK  on  24  May  2005. 

UGVs  are  currently  used  for  payload  delivery  (e.g.,  R-Gator,  Military  Robotic  Gator),  such  as  fire  brigade, 
medical  supplies,  hostage  scenarios  and  electronic  counter  measures  (ECM)  equipments.  Digney  [3]  provides 
a  review  of  research  at  DRDC-Suffield  on  UGVs  including  the  following: 

•  Unmanned  Scout  Vehicle  -  Tele-operated,  for  reconnaissance  operations; 

•  Caterpillar  D7  -  Both  telematic  and  on-board  human  operator,  for  earth  working;  and 

•  Improved  Landmine  Detection  Programme  (ILDP)  -  Telematic  control  for  landmine  detection. 

Digney  summarises  the  military  benefits  and  issues  of  UGVs  as  follows: 

•  Removal  of  soldiers  from  hazardous  and  hostile  environments. 

•  Robot  must  win  -  whatever  telematic,  shared  control  or  autonomous  system  is  fielded,  the  robot  must 
prevail  in  competitive  conflict. 

•  Hiding  complexity  from  operator  control  -  provide  autonomous  control  of  low  level  functions,  while 
human  controllers  supply  high  level  and  intuitive  directives. 

•  Amplified  use  of  manpower  -  field  more  vehicles  per  human  controller,  and  deploy  freed  personnel  to 
other  vital  roles. 

•  Persistent  attention  -  use  shared  control  and  persistent  search  for  detection  of  scene  changes  and  enlist 
human  assistance  in  change  classification,  to  mitigate  fatigue  and  inattentiveness. 

•  Lethal  force  control  -  automatic  control  of  lethal  force  is  not  permitted  by  current  ethical 
considerations.  Whenever  lethal  force  is  to  be  applied  from  an  UMV,  a  human  operator  must  be  in 
direct  control. 

•  Life  critical  operations  -  infra-red  image  classification  on  the  IDLP  mine  detection  vehicle  is 
currently  done  by  a  human  because  automatic  classification  performance  risks  machine  classification 
error  and  endangers  human  lives. 

•  Sacrificial  vehicles  -  unmanned  vehicles  losses  are  more  acceptable. 

•  Communications  silence  and  jamming  -  Urban  areas  exacerbate  problems  with  communications  links 
assurance,  jamming  of  communications  is  common,  communications  give  positions  and  intent  away, 
and  should  be  kept  to  a  minimum  and  performed  through  undetectable  means. 
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•  Acceptable  path  to  higher  autonomy  -  use  incremental  route  levels  of  autonomy,  supported  by 
incremental  verification,  a  demonstrable  safe  path,  with  progress  and  reliability  observed  by  the 
operator. 

Future  UGVs  are  envisioned  for  more  complex  and  hazardous  tasks,  such  as  casualty  evacuation  (REV-Robot 
Evacuation  Vehicle,  REX-Robotic  Extraction  Vehicle),  with  more  challenging  technical  requirements, 
complex  safety  issues,  and  potentially  high  levels  of  automation.  Future  UGVs  are  planned  with  improved 
capability  for  remote-controlled,  semi-autonomous  and  autonomous  operation  and  improved  mobility, 
flexibility  and  multi-functionality.  Enabling  technologies  include  ground  positioning  systems,  autonomous 
navigation,  automatic  collision  avoidance,  perception  and  navigation  sensors,  intricate  and  precise  positioning, 
and  artificial  intelligence  techniques  (e.g.,  hierarchical  learning,  adaptive  control,  neural  networks).  The  US 
Army’s  planned  Future  Combat  System  (FCS)  is  largely  built  on  UMV  concepts  (manned  vehicles  plus 
UAVs,  unattended  munitions  and  UGVs).  FCS  UGVs  comprise  the  Armed  Robotic  Vehicle  (ARV)  with 
assault  and  RSTA  (Reconnaissance,  Surveillance,  and  Target  Acquisition)  variants,  and  the  Multi-purpose 
Utility/Logistics  Equipment  (MULE)  for  countermine  and  transport. 

Looking  further  to  the  future,  the  US  Joint  Forces  Command’s  Project  Alpha  includes  “Unmanned  Effects: 
Taking  Humans  Out  of  the  Loop”.  This  seeks  to  explore  the  idea  of  that  autonomous,  networked  and 
integrated  robots  may  be  the  dominant  fighting  force  by  the  year  2025  (http://www.jfcom.mil/newslink/ 
storyarchive/2003/pa072903.htm  accessed  12/12/2005).  In  2003,  DARPA’s  “Centibots”  project  on  distributed 
robotics  looked  at  co-ordinated  deployment  of  groups  of  25  -  50  robots  in  advanced  surveillance  teams  for 
urban  missions,  including  area  surveying,  sharing  a  distributed  map  and  intruder  detection  (http://www.ai.sri. 
com/centibots  accessed  20/11/2005).  Several  advanced  research  programmes  are  using  biomimetics, 
the  engineering  of  a  process  or  system  that  mimics  biology,  to  investigate  behaviours  in  robots  that  emulate 
animals  such  as  self-healing  and  swarming  [2]. 


2. 1.3.2  Uninhabited  Underwater  Vehicles 

Uninhabited  underwater  vehicles  (UUVs)  have  been  deployed  by  NATO  Nations  in  a  variety  of  civilian  and 
military  roles.  Both  US  and  UK  have  active  research  and  development  programmes.  In  the  military 
underwater  domain,  more  than  others,  research  has  been  characterised  by  a  desire  for  a  direct  route  to 
behaviourally  simple,  but  fully  autonomous,  UUVs.  This  seems  mostly  due  to  the  military  importance  of  the 
littoral  zone  with  very  shallow  water  (VSW)  and  surf-zone  (SZ)  operations,  and  the  associated  problems  of 
difficult  underwater  acoustic  communications  (AComms). 

Carver  [4]  has  identified  a  variety  of  roles  for  UUVs  in  the  underwater  battle-space  derived  from  operational 
analysis,  surveys  of  likely  users  and  initial  concept  of  operations  studies: 

•  Mine  Countermeasures  (MCM); 

•  Environmental  Data  Gathering  (EDG); 

•  Rapid  Environmental  Assessment  (REA); 

•  Above  and  Below  Water  Intelligence  Gathering; 

•  Anti  Submarine  Warfare  (ASW)  -  Trainer; 

•  Expendable  Sensor  Deployment; 

•  Mobile  Signature  Range; 
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•  ASW  Bi-Static  Sonar  Operations;  and 

•  ASW  Track  and  Trail. 

Commercially  available  Remotely  Operated  Vehicles  (ROVs)  are  in  relative  abundance  for  deepwater 
operations.  Underwater  ROVs  tend  to  be  tethered  to  a  mother  ship  via  an  umbilical  cord,  supplying  power  and 
command  and  communications  links.  There  is  considerable  experience  using  underwater  ROVs  in  the  oil 
industry  for  offshore  support.  Specific  examples  include  the  Maridan  600  ROV  UUV  from  Denmark,  and  the 
Hugin  ROV  UUV  out  of  Norway.  In  addition,  UUVs  have  been  used  in  varied  environments  for  scientific 
work.  Academic  ROVs  include  the  Woods  Whole  Oceanographic  Institute’s  REMUS  (Remote  Environmental 
Monitoring  UnitS),  Florida  Atlantic  University’s  Morpheus,  Massachusetts  Institute  of  Technology’s  Odyssey 
and  Cetus  II,  and  Southampton  Oceanography  Centre’s  AutoSub  UUV.  In  the  UK,  Tiltman  [5]  reports  that  the 
combined  experience  of  industry  and  academia  is  currently  being  used  to  reduce  the  risk  to  UUV  procurement 
for  the  next  10  years  under  the  UK  MOD  Battlespace  Access  UUV  (BAUUV)  programme,  2003  -  2006. 

Tiltman  [5]  describes  how  the  UK  MARLIN  1995  -  2003  research  and  development  programme  was  based  on 
torpedo  technology  from  SPEARFISH  and  MK24’s.  MARLIN  was  designed  for  submarine  launch  and 
recovery,  with  a  unique  top  speed  of  15  kts.  Recovery  proved  particularly  difficult  and  requirements  for 
submarine  operations  have  subsequently  waned.  He  reports  that  the  Royal  Navy  currently  has  two  derivative 
REMUS  UUVs  in  active  service,  surveying  areas  in  and  around  ports,  harbours,  ship  lanes  and  landing  areas. 
The  US  Navy  is  developing  LMRS  (Long  Term  Mine  Reconnaissance  System),  a  mine  hunting  UUV.  These 
free  swimming  UUVs  tend  to  be  very  small,  of  limited  endurance,  with  a  single  specific  task,  such  as 
inspection  of  underwater  objects.  Tiltman  reports  that  joint  US/UK  GAMBIT  programme  is  developing  mine- 
warfare  UUV  sensors.  GAMBIT  uses  a  21”  UUV  built  by  Bluefln  Robotics  to  investigate  Synthetic  Aperture 
Sonar  (SAS),  mission  autonomy  and  navigation  systems.  He  believes  that  SAS  holds  the  possibility  of 
allowing  a  UUV  to  survey  a  boat  lane  in  one  sweep  at  4  kts.  This  will  significantly  change  mine  warfare 
tactics,  affect  the  whole  amphibious  operation,  and  significantly  reduce  the  risk  to  military  personnel. 

Posey  [6]  argues  that  MCM  is  a  key  role  for  UUVs,  e.g.,  RAUVER,  REMUS.  The  sea  mine  remains  a 
powerful  and  cost  effective  asymmetric  threat  of  significant  concern  to  the  maritime  forces.  Posey  believes 
that  current  dedicated  MCM  capabilities  will  not  satisfy  the  requirements  of  the  future  battlespace.  He  reports 
that  this  is  because  current  MCM  capabilities  are  limited  by  lengthy  timelines  for  surface  assets  to  arrive  in 
theatre,  inadequate  integration  of  assets,  minimal  reconnaissance  means,  and  operational  pauses  created  by  the 
slow,  deliberate  nature  of  MCM  operations. 

Waters  and  Taylor  [7]  report  that  the  US  Navy  envisage  the  littoral  zone  to  be  the  most  important  for  UUV 
operations,  from  MCM  prior  to  a  naval  assault,  through  coastal  and  channel  mapping  via  sonar,  to  deploying 
sonars  near  enemy  naval  installations  to  track  asset  movement  and  even  kill  them  with  torpedoes.  MCM  and 
Explosives  Ordnance  Disposal  are  key  operational  tasks  for  UUVs  in  VSW/SZ  operations.  Blackburn  et  al.  [8] 
report  that  a  VSW  MCM  detachment  comprises  70  personnel:  40  are  in  operations,  mostly  diver  qualified, 
18  go  into  the  water,  21  service  and  control  dolphins,  and  6  will  operate  UUVs.  The  dolphins  locate  the  mines 
using  endogenous  sonar,  then  drop  pingers  to  tag  locations.  For  SZ  operations,  brute  force  neutralisation  is 
used  by  laying  down  a  blanket  of  charges  for  in-stride  breaching. 

Blackburn  et  al.  [8]  note  that  the  closer  the  UUV  is  driven  to  the  beach,  the  greater  the  sensing,  navigation, 
communications  and  control  problems  become.  With  ROV,  in  both  VSW/SZ  and  deep  water  operations, 
there  is  the  possibility  of  increased  difficulty  of  task  execution  with  tele-operated  ROV  control  from  a 
distance.  This  is  particularly  problematic  if  information  is  changing  quickly,  or  restricted,  as  it  often  is  in  the 
complex  terrestrial  environment  and  near  beach  underwater  environment. 
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A  major  factor  governing  UUV  operations  is  the  availability  of  power.  Strong  currents  and  water  turbulence 
can  make  station  keeping  difficult  to  achieve  and  drain  power.  Walters  and  Taylor  [7]  describe  how  retaining 
and  managing  power  is  an  important  UUV  task,  and  a  strong  candidate  for  automation.  The  power  source  will 
also  have  to  be  wholly  internal  to  the  vehicle.  This  is  currently  based  on  batteries,  although  it  is  likely  that  fuel 
cells  will  replace  these  as  the  technology  improves.  Even  the  latter  will  only  yield  a  useful  energy  output  of 
about  400Wh  per  kg,  with  the  potential  to  (possibly)  double  this  figure  within  the  next  5  years.  With  current 
UUVs,  such  as  the  USS  Manta,  requiring  up  to  50  kW  for  propulsion  alone,  plus  several  kW  more  for  sensor 
operation  (e.g.,  sonar),  the  size  of  the  problem  of  supplying  sufficient  power  to  allow  the  performance  of  any 
kind  of  mission  becomes  apparent.  The  addressing  of  this  power  management  problem,  which  is  usually 
denoted  by  HOTEL,  boils  down  to  answering  the  question  “can  I  do  the  mission  and  return  to  my  recovery 
point  on  my  power  reserves?”  This  base-lining  of  the  projected  energy  consumption  for  the  whole  mission, 
continually  updated  during  the  mission,  underpins  every  other  assessment  and  decision  made  during  the 
mission. 

In  wider  concepts,  UUV  missions  may  last  for  days.  Alternatives  include  large  UUVs  carrying  a  variety  of 
sensors  and  deploying  sensor  arrays,  and  smaller  vehicles  firing  torpedoes  [4].  This  plethora  of  UUV  roles 
leads  to  the  concept  of  adopting  a  modular  design,  where  different  operational  modules  can  be  fitted, 
so  providing  ‘swing  roles’  from  a  baseline  vehicle.  Carver  notes  that  a  common  thread  emerges.  To  achieve 
the  desired  range  and  endurance,  and  to  deliver  the  required  capability,  UUVs  are  becoming  larger, 
embodying  more  autonomy  and  developing  into  complex  ‘systems  of  systems.’  As  with  current  UAVs,  UUVs 
will  soon  fire  weapons  on  command,  requiring  reliable  secure  underwater  communications  systems.  Further 
into  the  future,  the  ultimate  ‘leap  of  faith’  will  be  to  permit  weapons  to  be  released  autonomously. 

Looking  into  the  future,  iRobot  have  a  DART  biomimetic  programme  for  small,  autonomous  UUVs 
that  emulate  the  efficiency,  acceleration  and  manoeuvrability  of  fish.  Also,  they  have  proposed  ALUV 
(Ariel  Autonomous  Legged  Underwater  Vehicle),  a  crab-like  robot,  for  mine  and  obstacle  neutralisation. 
ALUV  would  secure  itself  to  the  mine  and  await  a  detonation  signal,  or  deposit  an  explosive. 


2. 1.3.3  Uninhabited  Air  Vehicles 

2. 1.3. 3.1  Benefits  of  UAVs 

In  2004,  NATO  Industrial  Advisory  Group  (NIAG)  Study  Group  75  reported  an  overview  of  status  of  UAV 
technology  from  a  NATO  industry  perspective  [9].  In  general,  it  is  believed  that  the  ability  of  UAVs  to 
perform  their  missions  with  autonomous  capabilities  will  be  a  major  step  towards  achieving  flexible,  efficient 
and  interoperable  military  (including  combat)  operations.  Broadly,  the  advantages  of  UAVs  are  believed  to 
include  the  following: 

•  Reduced  risk  to  humans; 

•  Optimised  operator  performance; 

•  Reduced  training  requirements; 

•  Improved  contingency  management; 

•  Reduced  data  link  demands;  and 

•  Increased  operational  flexibility. 

Air  Chief  Marshall  Burridge  [10]  concisely  summarises  the  strengths  and  challenges  of  UAVs  as  follows: 
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Strengths 

•  Dealing  well  with  3D  tasks  -  dull,  dirty  and  dangerous; 

•  Potential  response  at  a  number  of  levels,  ranging  from  resolving  tactical  firepower  problems  to 
providing  commander’s  strategic  critical  information  requirements; 

•  Ease  of  re-tasking; 

•  Increase  stand-off  ranges  for  kinetic,  and  non-kinetic  or  cognitive  attack;  and 

•  Persistence. 

Challenges 

•  Interoperability  of  systems; 

•  Vulnerability; 

•  Limited  capacity  to  address  a  wide  surveillance; 

•  Insatiable  demand  for  bandwidth;  and 

•  Inability  to  deal  with  ambiguity  in  the  same  way  as  manned  aircraft. 

2. 1.3. 3. 2  Classes  of  UAV 

A  parallel  activity  has  been  conducted  under  the  NATO  Systems  Concepts  and  Integration  Panel  (SCI)  Panel, 
namely  NATO  SCI- 124  Task  Group,  Architecture  for  the  Integration  of  Manned  and  Unmanned  Air  Vehicles. 
Close  liaison  between  SCI-124  and  HFM-018  was  conducted  principally  by  UK,  US  and  GE  TG  members. 
This  interaction  has  been  particularly  beneficial  for  HFM-018  in  identifying  a  range  of  UAV  assumptions. 
For  the  purposes  of  the  present  report,  basic  information  on  C2  architecture,  tasks  and  classes  of  UAVs  are 
taken  directly  from  the  Final  Report  of  NATO  SCI-124  Task  Group  [11]. 

Six  classes  of  UAV  are  identified  by  SCI-124  with  role-specific  platform  characteristics  and  control 
requirements.  These  are  summarised  in  Table  2-1  below  from  the  SCI-124  Final  Report. 


Table  2-1:  Summary  of  UAV  Classes,  Roles  and  Control 


Class 

Description 

Role 

Control 

CML-UAV 

Like  cmise  missile 

Subsonic,  reusable 

Deep  penetration,  low  altitude, 
terrain  following 

Independent  operation  or  in 
support  of  manned  strike  aircraft 

Pre  strike  recce 

Post  strike  recce  (BDA) 
Target  identification 

Target  verification 

Third  party  target 
illumination 

Automatic: 

•  Flying 

•  Pre-programmed: 

•  Flight  path 

•  Sensor  operation 

UCS: 

•  Flight,  mission 

•  Update  way  points 

•  Payload 
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Class 

Description 

Role 

Control 

CAL-UAV 

(UCAV) 

Like  combat  aircraft 

Deep  penetrating  stealthy 
ground  attack 

Examples:  Joint  Unmanned 

Aerial  Combat  System 
(J-UCAS)  demonstrators: 

•  Boeing  X-45 

•  Northrop  Grumman 

X-47  Pegasus 

Strike 

SEAD 

DEAD 

Recce 

Surveillance 

Automatic: 

•  Flying 

•  Pre-programmed  with 
human-in-the-loop, 
or  autonomous: 

•  Target  acquisition 

•  Target  verification 

•  Weapon  release 

UCS  (+  CAOC,  AEWC): 

•  Flight,  mission 

•  Update  way  points 

•  Payload 

HALE-UAV 

High  altitude,  Long  endurance 
Examples: 

•  Global  Hawk 

•  General  Atomics  Predator  B 

Stand  off 

Strategic 

Over  target  area 

Surveillance 

Recce  (IMINT,  SIGINT) 
Target  acquisition 

Stand-off  jamming 

Automatic: 

•  Flying 

•  Pre-programmed: 

•  Flight  path 

•  Sensor  operation 

UCS: 

•  Takeoff,  landing 

•  Flight,  mission 

•  Update  way  points 

•  Payload 

MALE-UAV 

Medium  altitude  -  up  to  10  km 
Long  endurance,  Medium  speed 
Examples: 

•  Predator 

•  Hunter 

•  Heron 

•  Watchkeeper 

Stand  off 

Tactical 

Over  target  area 

Surveillance 

Recce  (IMINT,  SIGINT) 
Target  acquisition 

Stand-off  jamming 

Automatic: 

•  Flying 

•  Pre-programmed: 

•  Flight  path 

•  Sensor  operation 

UCS: 

•  Takeoff,  landing 

•  Flight,  mission 

•  Update  way  points 

•  Payload 

VTOL-UAV 

Future,  vertical  take-off  and 
landing 

Forward  base  operations  range 
(radius)  500  km  fast 
(>300  km/h)  low 
(nap-of-the-earth) 

Combat  search  and  rescue 
(CSAR) 

Casualty  variant 
(armoured  cabin) 

Escort  protection  variant: 

•  Remotely  controlled 
machine  guns 

•  IR-/TV-cameras 

UCS: 

•  Monitor  and  control 
flight,  mission 

•  Control/operate  escort 
weapons 
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Class 

Description 

Role 

Control 

FAL-UAV 

Future,  like  fighter  aircraft 

Air  combat 

Highly  agile 

Supersonic 

Air  combat,  highly 
reactive  against  hostile 
manned  fighter  aircraft 
and  enemy’s  UAVs 

Automatic:  High  levels  of 
autonomy  (2020): 

•  Pre-programmed 
ingress  and  egress 

•  Air-target  acquisition 

•  Initiation  of  combat 
flight  manoeuvres 

•  Tactical  manoeuvres 
based  on  human 
intelligence 

•  Weapon  selection 

•  Weapon  release 

•  Fire  control 

UCS  (+  CAOC  ,AEWC): 

•  Flight,  mission 

•  Update  way  points 

•  Monitor  autonomous 
operation 

•  Intermpt  weapon 
release 

Small  portable  reusable 

Support  for  ground  forces 

UCS: 

Takeoff  weight 

Beyond  visible  range 

•  One  person  operation 

<  5  kg  Range  <  10  km 

(BVR)  capability 

•  Mobile 

Mini-UAV 

Altitude  500  m 

Reconnaissance 

•  Laptop 

Endurance  30  minutes. 

Example:  Dragon  Eye 

Target  acquisition 

•  Hand-bag  portable 

2. 1.3. 3. 3  UAV  Control  Stations 

All  the  UAV  classes  identified  are  operating  automatically  with  pre-programmed  flight  paths  and 
pre-programmed  payload/sensor  control.  The  UAV  operators  can  change  the  way  points  and  the  sensor 
control  functions  in  the  tactical  UAV  Control  Stations  (UCS)  during  the  flight.  Furthermore,  the  UAV  and 
payload  control  can  be  handed  over  to  the  CAOC  or  other  sea-,  land-  or  air-based  tactical  UCSs  at  any  time 
during  the  flight.  Autonomy  levels  define  the  UAV’s  level  of  control  and  hence  the  required  commands  to 
control  it.  At  present,  UAVs  need  a  UCS  with  an  operator.  The  location  of  the  UCS  is  critical  for  future 
CONOPS.  The  UCS  location  could  be  remote,  or  located  in  an  accompanying  aircraft.  The  functions  of  the 
UCS  are  summarised  by  SCI-124  as  follows: 

•  C2  of  air  vehicle  and  payload  (including  weapons); 

•  Possibly  (limited)  processing,  display  and  exploitation  of  sensor  data; 

•  Communications  with  Air  Control  Centre  (ACC);  and 

•  Dissemination  of  UAV  sensor  data  to  users. 
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The  UCS  can  be  land,  sea  or  air-based.  UCS  can  control  many  UAVs.  One  UAV  can  be  controlled  (over  time) 
by  many  UCS  both  on  the  ground  as  airborne,  but  never  at  the  same  time. 

Currently,  one  UCS  is  needed  to  operate  one  UAV.  SCI- 124  believes  that  future  UAV  systems  will  have  the 
ability  to  control  several  vehicles  with  one  UCS.  Control  of  a  UAV  or  payload  may  be  passed  from  one  UCS 
to  another.  Control  handover  and  associated  operator  workload  issues  will  need  to  be  considered  in  deriving 
the  integration  and  C2  requirements  for  a  manned  and  unmanned  aircraft  mix. 

2. 1.3. 3. 4  UAV  Autonomous  Control  Levels 

The  DoD  UAV  Roadmap  2002  defines  10  levels  of  autonomous  control  (ACL).  These  provide  a  quantification 
of  the  wide  range  of  UAVs  that  will  affect  UCS  HF  requirements: 

•  ACL  10  -  Fully  autonomous  swarms; 

•  ACL  09  -  Groups  strategic  goals; 

•  ACL  08  -  Distributed  control; 

•  ACL  07  -  Group  tactical  goals; 

•  ACL  06  -  Group  tactical  re-plan; 

•  ACL  05  -  Group  coordination; 

•  ACL  04  -  Onboard  route  re-plan; 

•  ACL  03  -  Adapt  to  failures  and  flight  conditions; 

•  ACL  02  -  Real  time  health  diagnosis;  and 

•  ACL  01  -  Remotely  guided. 

The  Predator  UAV  is  represented  as  at  ACL  2,  Global  Hawk  UAV  as  at  ACL  2.5.  ACL  5  should  be  reached 
by  2010. 

SCI- 124  believes  that  a  UAV  must  respond  to  commands  in  a  timely  way,  similar  to  manned  aircraft.  This  is 
achieved  by  the  combination  of  operator  intervention  and  ACL.  It  is  assumed  that  by  definition,  at  low  UAV 
ACL,  there  will  be  much  greater  operator  intervention,  and  at  high  ACL,  only  little  operator  intervention  will 
be  required.  Implementation  of  this  vision  of  increasing  delegation  of  UAV  control  to  automation  together 
with  a  managed  reduction  in  operator  involvement  per  UAV  presents  major  HF  challenges. 

Current  UAVs  have  limited  sense  and  avoid  capability.  SCI- 124  have  concluded  that  enhancement  of  current 
UAV  sense  and  avoid  capability  will  be  essential  in  order  to  mix  manned  and  unmanned  aircraft  in  the  same 
package.  They  note  that  future  UAV  systems  will  require  sensors  that  detect  traffic  and  robust  automatic 
collision  avoidance  systems  that  are  able  to  react  in  a  manner  similar  to  a  manned  aircraft. 

2.1.3.3.5  C2/ISR  UAVs 

Looking  to  the  future  requirement  for  UAVs  in  the  US  Army,  Curran  [12]  describes  the  future  core  capabilities 
for  command  and  control  (C2),  as  well  as  intelligence,  surveillance  and  reconnaissance  (ISR).  He  states  that 
UAVs  are  becoming  increasingly  important  in  enabling  ground  forces  “to  see  first ,  understand  first ,  act  first, 
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and  then  finish  decisively  He  believes  that  they  provide  critical  information  without  jeopardising  or  risking 
lives.  He  notes  that  as  range,  altitude  and  loiter  time  increase  UAVs  are  providing  “the  eyes”  to  support  line-of- 
sight  and  beyond-line-of-sight  reconnaissance,  fires  and  over-watch.  As  a  result,  Curran  describes  that  this 
support  enables  rapid  movement,  target  identification  and  engagement,  and  enhanced  battle  damage  assessment, 
giving  commanders  greater  understanding  of  the  effects  of  their  combat  operations.  In  future,  Curran  believes 
that  all  classes  of  UAVs  will  have  the  following  core  capabilities: 

•  Networked  systems-of-systems  using  the  Future  Combat  System  (FCS)  common  operating 
environment,  common  computers,  software,  sensors  and  battle  command  communications  management. 

•  Embedded  autonomous  flight  control  and  navigation,  and  safe  flight  protocols. 

•  Unprecedented  reliability,  maintainability  and  operational  availability  reducing  the  quantity  of 
logistics  support. 

•  Condition-based  maintenance  with  sophisticated  prognosis  and  diagnosis  data  networked  into  the 
Brigade  Combat  Team. 

•  A  reusable  platform  durable  against  environmental  effects. 

•  Power  sources  readily  available  within  the  BCT  and  compatible  fuel/power  cells. 

•  An  anti-tamper  capability. 

•  A  means  to  sense  and  report  a  personnel  and  vehicle  presence  night  and  day. 

•  The  ability  to  report  target  and  platform  locations. 

•  The  capability  to  detect  and  report  location  and  direction  of  threat  systems  acquiring,  or  targeting,  the 
UAV  platform. 

2.133.6  Combat  UAVs 

Of  particular  significance  in  looking  to  the  battlefield  of  the  future,  the  Joint  Unmanned  Combat  System 
(J-UCAS)  is  a  joint  effort  between  the  Defence  Advanced  Research  Projects  Agency  (DARPA),  the  US  Air 
Force  and  the  US  Navy.  Francis  and  Hirschberg  [13]  describe  how  the  J-UCAS  programme  seeks  to  exploit 
the  potential  of  a  networked  system  of  high  performance,  weapon-carrying  unmanned  aircraft  with  the  ability 
to  penetrate  and  persist  deep  within  the  enemy  territory.  J-UCAS  is  intended  to  develop  the  capability  for 
unmanned  aircraft  to  cooperatively  locate  and  attack  an  integrated  air  defence  system  (IADS)  without  risking 
the  lives  of  pilots.  Francis  and  Hirschberg  report  that  the  high  levels  of  autonomy  planned  for  J-UCAS  are 
crucial  for  operating  world-wide,  independently  of  degraded  communications.  Autonomy  is  needed  so  that 
degraded  communications,  whether  caused  by  sunspots  or  jamming,  must  not  impair  the  aircraft  functionality 
or  the  system’s  ability  to  complete  missions  within  the  assigned  rules  of  ROE.  The  example  ROE  given  is  the 
use  of  force  only  authorised  by  the  human  operator.  According  to  Francis  and  Hirschberg,  the  intention  is  for 
J-UCAS  to  move  from  the  current  crewing  norm  involving  multiple  operators  controlling  a  single  UAV,  to  a 
new  crewing  paradigm  with  multiple  aircraft  being  controlled  by  a  relatively  small  number  of  operators. 
They  state  that  the  vision  is  that  this  will  be  achieved  with  the  operators’  tasking  optimised  for  workload  and 
mission-critical  needs.  J-UCAS  will  collaborate  with  other  J-UCAS  aircraft  and  additional  assets  to  enhance 
overall  SA  and  improve  the  speed  and  precision  of  geo-location,  identification,  tracking,  and  attack  and 
assessment  of  targets.  Operating  at  the  speed  and  altitude  of  commercial  air  traffic,  J-UCAS  will  have  a  “sense 
and  avoid”  capability  to  be  able  to  operate  routinely  in  the  global  air  traffic  management  system  together  with 
an  automated  aerial  refuelling  capability.  Collaboration  with  the  UK  Ministry  of  Defence  has  been  established 
to  investigate  the  military  benefits  and  interoperability  issues  in  future  coalition  operations  through 
experimentation  and  distributed  real-time  simulation. 
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2. 1.3. 3. 7  NATO  Allied  Ground  Surveillance  System 

Currently,  NATO  has  a  requirement  for  an  Allied  Ground  Surveillance  (AGS)  system  in  support  of  their 
activities.  AGS  is  a  method  to  support  Peace  Keeping  and  other  military  operations.  It  requires  a  radar  system 
capable  of  simultaneous  Synthetic  Aperture  Radar  (SAR)  mode  and  Moving  Target  Indicator  (MTI)  mode. 
These  modes  must  perform  at  long  stand-off  distances  and  in  high  resolution.  The  system  must  within  a  large 
area  of  interest  image  any  details  with  the  SAR  mode;  in  the  MTI  mode  it  must  detect,  track  and  classify 
moving  targets.  The  AGS  is  primarily  designed  for  military  use,  but  also  proves  efficient  in  border  control 
activities. 

Five  European  countries  (Germany,  France,  Italy,  Spain,  and  The  Netherlands)  have  responded  to  a  NATO 
request  and  proposed  a  demonstrator,  the  Stand  Off  Surveillance  and  Target  Acquisition  System  (SOSTAR) 
[14].  The  SOSTAR  demonstrator  is  intended  to  perform  all  the  required  functions  of  the  full  scale  model, 
including  the  simultaneous  interleaved  operation  of  SAR  and  MTI,  but  has  a  small  antenna  size  to  reduce  cost. 
The  system  could  be  used  as  a  basis  for  an  AGS  system  on  UAV’s  and  on  small  aircraft.  The  final  version  of 
SOSTAR  is  foreseen  to  have  a  scalable  antenna  and  is  suitable  to  be  installed  on  NH-90  helicopter  platforms 
and  High  Altitude  Long  Endurance  (HALE)  UAV’s.  The  system  may  become  available  by  the  end  of  the 
decade.  The  system  is  set  up  in  such  a  way  that  that  it  can  not  only  fulfil  NATO’s  requirements,  but  also  any 
upcoming  national  requirement.  Although  AGS  systems  are  still  in  their  childhood,  the  first  valuable 
experiences  are  there. 

2. 1.3. 3. 8  Future  Autonomous  UAVs 

NIAG  SG-75  reported  the  following  conclusions  regarding  the  development  and  operation  of  autonomous 
UAVs  [10]: 

•  Operating  future  autonomous  UAV  systems  within  a  Network  Centric  Warfare  (NCW)  environment 
will  be  enable  the  human  to  assume  the  role  of  system  manager,  rather  than  system  operator,  and  that 
will  be  advantageous  for  a  variety  of  missions. 

•  The  development  of  autonomy  for  UAV  systems  is  feasible.  It  will  depend  not  just  upon  the 
development  of  technologies  identified  in  this  study,  but  also  upon  doctrinal  and  policy  considerations, 
particularly  during  operations  with  combat  UAVs. 

•  Autonomy  can  be  realised  incrementally  with  progressively  increasing  levels  of  autonomy. 
The  technologies  required  to  perform  UAV  autonomous  operations  have  been  identified,  emphasis 
being  placed  on  technology  solutions  that  would  be  possible  within  5-10  years. 

•  A  number  of  these  technologies  were  identified  to  be  critical,  but  are  not  yet  at  a  Technology 
Readiness  Level  (TRL)  allowing  design  and  manufacture. 

•  The  level  of  autonomy  will  depend  upon  the  capabilities  required  of  an  autonomous  UAV  system, 
which  is  a  function  of  the  missions  the  system  must  perform. 

•  Technologies  duplicating  flight  crew  functions  are  critical  for  developing  autonomy. 

•  Development  cost  can  be  roughly  estimated  by  the  TRL  for  a  technology. 

•  Cost  and  risk  for  developing  technologies  to  meet  autonomous  capabilities  are  dependent  not  only 
upon  the  level  of  maturity  of  the  technology,  but  also  upon  mission  and  system  requirements. 

•  Decision  making,  modelling,  learning,  and  attack  planning  have  the  highest  risk/highest  cost  associated 
with  their  development  in  UAV  systems,  but  they  also  support  the  greatest  number  of  autonomous 
capabilities. 
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•  Technologies  related  to  status  assessment,  weapons  engagement,  prediction,  and  mission  plan  update 
is  relatively  lower  risk  and  lower  cost. 

•  A  number  of  technologies  have  been  identified  which  are  not  uniquely  critical,  but  contribute  to 
different  technical  solutions  for  autonomous  operations.  It  is  therefore  not  necessary  to  develop  all  of 
these  enabling  technologies. 

•  An  autonomous  UAV  operating  within  a  network  centric  environment  needs  to  behave  with  the  same 
reliability  and  effectiveness  as  manned  elements  and  ideally,  should  not  require  special  handling. 

•  Due  to  the  low  TRL  of  recognising  and  comprehending  natural  speech,  it  is  considered  improbable  that 
autonomous  verbal  interaction  with  civil  air  traffic  control  (ATC)  can  be  developed  to  the  level  of 
reliability  and  integrity  demanded  by  the  Authorities.  Until  ATC  interactions  will  routinely  done  by  data 
link,  autonomous  crossing  of  civil  airspace  under  ATC  control  is  therefore  not  considered  feasible. 

•  The  proposed  Technology  Demonstration  Programmes  (TDPs)  are  an  important  step  in  the 
development  of  autonomous  UAV  capacities. 

•  It  is  most  effective  to  conduct  TDPs  that  improve  the  TRL  level  of  technologies  from  TRL  5  to 
TRL  6. 

2. 1.3.4  Summary  of  UMV  Missions,  Roles  and  Tasks 

Based  on  the  foregoing,  UMV  roles  and  tasks  can  be  broadly  distinguished  according  to  vehicle  environments 
(UUV,  UGV  and  UAV)  and  missions  (ISR,  Combat  and  Support  Operations).  These  are  summarised  in 
Table  2-2. 


2-20 


RTO-TR-HFM-078 


dMNATO 
WP  I  OTAN 


MILITARY  RELEVANCE 


Table  2-2:  Summary  of  UMV  Missions  and  Environments 


Missions 

Uninhabited  Ground 
Vehicles  -  UGVs 

Uninhabited 

Underwater  Vehicles  - 
UUVs 

Uninhabited  Air 

Vehicles  -  UAVs 

Intelligence, 

Surveillance, 

Reconnaissance 

Reconnaissance,  surveillance 
and  target  acquisition 

Searching  underground  in 
caves,  culverts,  drains  and 
man  holes 

Searching  collapsed 
buildings 

Environmental  data 
gathering 

Rapid  environmental 
assessment 

Above  and  below  water 
intelligence  gathering 

Expendable  sensor 
deployment 

Deployment  of  sonars  near 
enemy  naval  installations 

ASW  Bi-Static  sonar 
operations 

ASW  Track  and  trail 

Track  asset  movement 

Surveying  areas  in  and 
around  ports,  harbours,  ship 
lanes  and  landing  areas 

Coastal  and  channel  sonar 
mapping 

Pre-strike  reconnaissance 

Post-strike  reconnaissance 

Battle  damage  assessment 
Stand-off  operations 

Over  target  area  operations 

Support  for  ground  forces 

Beyond  visible  range  (BVR) 
capability 

Support  line-of-sight  and 

beyond-line-of-sight 

reconnaissance 

Stand  off  ground  Surveillance 

Report  target  and  platform 
locations 

Sense  and  report  a  personnel  and 
vehicle  presence  night  and  day 

Detect  and  report  location  and 
direction  of  threat  systems 
acquiring,  or  targeting,  the 

UAV  platform 

Combat 

Operations 

Land  mine  detection  and 
neutralisation 

Explosive  ordnance 
clearance  and  disposal 
(EOD/IED) 

Urban  warfare  operations  in 
confined  spaces 

Breaching,  earth  working 
Casualty  recovery 

Deployment  of  weapons  or 
obscurants 

Firing  of  rifles,  machine 
guns,  grenade  launchers  and 
rockets 

Deployment  as  a  decoy 
target 

Hostage  scenarios 

Deployment  of  electronic 
counter  measures 

Mine  countermeasures 

Mine  hunting 

Mine  counter  measures  prior 
to  a  naval  assault 

Explosives  ordnance 
disposal 

Kill  enemy  assets  with 
torpedoes 

Mine  neutralisation 

Obstacle  neutralisation 

Target  identification 

Target  verification 

Third  party  target  illumination 

Strike 

Suppression  of  enemy  air 
defences 

Destruction  of  enemy  air 
defences 

Stand  off  jamming 

Stand  off  target  acquisition 

Air  combat,  highly  reactive 
against  hostile  manned  fighter 
aircraft  and  enemy’s  UAVs 

Line-of-sight  and 
beyond-line-of-sight  fires 
and  over- watch 
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Missions 

Uninhabited  Ground 
Vehicles  -  UGVs 

Uninhabited 

Underwater  Vehicles  - 
UUVs 

Uninhabited  Air 

Vehicles  -  UAVs 

Support 

Operations 

Operations  in  contaminated 
areas  and  contamination 

assessment 

Detection  and  analysis  of 
chemical,  nuclear  and 
biological  assets 

Fire  fighting 

Logistics  -  delivery  to  the 
battlefield  of  munitions,  fuel, 
parts,  food  and  water 

Security  and  patrol 

Deployment  as  a  mobile 
communications  link 

Deployment  as  a  mobile 
power  supply 

Mobile  signature  range 

Anti  submarine  warfare  - 
Trainer 

VTOL  combat  search  and  rescue 
(CSAR) 

CSAR  casualty  transport 
(armoured  cabin) 

CSAR  escort  protection  with 
remotely  controlled  machine 
guns  and  IR-/TV  cameras 
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2.1.4  Command  and  Control 

A  recent  analysis  of  C2  taken  from  an  HF  perspective,  holds  that  C2  is  rooted  in  the  authority,  responsibility 
and  will  of  the  human  commander,  including  the  human  use  of  C2  systems  [1,  see  also  2,3].  C2  is  traditionally 
viewed  from  a  technology  perspective  as  a  set  of  networking,  information  processing,  and  computational 
problems.  As  an  organisational  structure,  C2  reflects  the  chain  of  command  and  concerns  the  delegation  of 
authority  and  the  expectation  of  responsibility  and  accountability.  The  chain  of  command  is  an  important 
mechanism  for  accomplishing  the  transmission  of  commander’s  will.  It  is  a  process  of  C2  by  which 
commanders  exercise  authority  over  forces  to  plan  and  direct  operations.  A  key  conclusion  from  this  analysis 
is  that  C2  is  fundamentally  a  human-centric  undertaking,  shaped  by  the  nature  of  the  military  mission,  by  the 
security  environment,  and  by  the  fundamental  culture  of  the  military  itself.  The  concept  of  C2  centres  around 
the  actions  of  a  commander  who  has  been  designated  the  authority  to  carry  out  those  actions  in  respect  of 
resources  and  subordinates,  and  who  is  expected  to  take  responsibility  for  proper  use  of  that  authority.  In  this 
perspective,  UMV  technologies  are  command  support  tools  and  systems,  and  a  vital  aspect  of  C2.  Thus,  the 
underlying  purpose  of  technology,  such  as  UMVs,  is  to  extend  human  capability  -  “whether  it  is  the  physical 
ability  to  act  in  the  operational  environment,  the  intellectual  ability  to  make  better  decisions  faster,  or  the 
interpersonal  ability  to  communicate  with  distributed  subordinates  and  security  partners”. 
The  challenge  is  to  ensure  that  UMV  technologies  are  properly  integrated  with  all  aspects  of  C2  systems, 
but  in  particular,  the  human-centric  nature  of  C2,  so  as  to  support  efficiently  and  effectively  the  commander’s 
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authority,  and  expectations  of  responsibility  and  accountability.  For  further  detailed  discussion  of  HF  aspects 
of  UMV  C2  requirements  and  broader  system-of-system  issues,  see  Chapter  4. 

2. 1.4.1  NATO  Air  Command  and  Control  System 

The  C2  of  NATO  air  forces  including  manned  and  uninhabited  aircraft  and  missiles,  C2  centres, 
communications  and  sensors  (and  Command,  Control,  Communication,  Computers,  Intelligence,  Surveillance 
and  Reconnaissance  -  C4ISR)  is  performed  by  NACCS.  NATO  SCI- 124  has  considered  the  importance  of 
NACCS  for  C2  of  UAVs  [4].  UAV  systems  are  currently  not  part  of  the  NACCS  and  are  not  managed  as  part 
of  a  manned  strike  package.  In  future  systems,  SCI- 124  has  concluded  that  UAVs  will  need  to  be  closely 
integrated  in  the  NACCS  with  manned  aircraft  if  they  are  to  operate  in  the  same  force  package.  Therefore, 
integration  with  NACCS  needs  to  be  considered  in  the  broad  concept  of  use  for  UAV  systems. 

NACCS  is  a  hierarchical  structure  of  headquarters  with  the  operations  centres  of  the  System  Command  (SC) 
at  the  top  and  operations  centres  of  the  executing  units  at  the  bottom  (Figure  2-1).  This  hierarchical  structure 
consists  of  three  levels:  a  planning  level,  in  which  the  commanders  deal  primarily  with  the  allocation  of 
targets,  objectives,  resources  and  areas  of  responsibility  to  subordinate  commanders;  a  tasking  level,  in  which 
the  commanders  deal  primarily  with  the  assignment  of  tasks  to  resources  for  execution;  and  an  execution 
level,  in  which  the  commanders  deal  with  the  control  and  management  of  the  execution  of  assigned  tasks. 
Four  entity  types  Combat  Air  Operations  Centre  (CAOC),  Air  Control  Centre  (ACC),  Rapid  Air  Picture 
(RAP)  Production  Centre  (RPC)  and  Sensor  Fusion  Post  (SFP)  constitute  the  “core”  of  NACCS.  Integration  of 
UAV  operations  into  NACCS  has  impact  on  UAV  operations  planning,  tasking,  tactical  control,  detection  and 
tracking,  identification,  in  addition  to  data  transmission,  data  dissemination,  and  airspace  management  and 
traffic  deconfliction. 
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Figure  2-1 :  NATO  Air  Command  and  Control  System  -  NACCS. 
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NATO  SCI- 124  have  identified  that  NACCS  will  disseminate  tasking  information  to  all  UAV  operating 
locations.  UAV  sorties  must  be  included  in  the  Air  Task  Order  (ATO)  along  with  manned  aircraft  sorties  for 
maximum  combined  effectiveness.  NACCS  will  support  coordination  of  UAV  operations  with  manned 
aircraft  and  ground-based  defences,  including  real-time  C2  of  UAV  in-flight  activities,  and  a  command, 
control  and  communications  interface  between  the  NACCS  entity  and  the  UAV  Control  Station  (UCS). 


2. 1.4.2  Interoperability 

Interoperability  is  needed  in  Combined/Joint  services  operations  to  provide  close  co-ordination,  the  ability  to 
task  quickly  available  assets,  and  the  rapid  dissemination  of  resultant  information  at  different  command 
echelons.  Current  and  legacy  UAV  systems  are  not  interoperable.  They  do  not  have  standard  interfaces 
between  the  system  elements,  nor  with  C2  and  ISR  systems.  NATO  STANAG  4586  [5]  seeks  to  define  the 
standards  for  key  system  interfaces  and  functions  in  the  UCS  to  communicate  with  different  UAVs  and  their 
payloads,  as  well  as  different  C2/ISR  systems,  to  achieve  levels  of  interoperability  required  by  the  system’s 
CONOPS.  STANAG  4586  includes  a  human-computer  interface  (HCI)  style  guide  on  best-practice.  Multiple 
levels  of  interoperability  are  feasible  among  different  UAV  systems.  Maximum  operational  flexibility  can  be 
achieved  if  the  UAV  systems  support  the  following  levels  of  UAV  system  interoperability  identified  by 
STANAG  4586,  based  on  studies  performed  by  AC/141  (PG/35)  and  NATO  Industrial  Advisory  Group 
(NIAG),  Sub  Group  (SG)  53: 

•  Level  1 :  Indirect  receipt  of  secondary  imagery  and/or  data. 

•  Level  2:  Direct  receipt  of  payload  data  by  a  UCS;  where  “direct”  covers  reception  of  the  UAV 
payload  data  by  the  UCS  when  it  has  direct  line-of-sight  with  the  UAV  or  a  relay  device  which  has 
direct  line-of-sight  with  the  UAV. 

•  Level  3:  Level  2  interoperability  plus  control  of  the  UAV  payload  by  a  UCS. 

•  Level  4:  Level  3  interoperability  plus  UAV  flight  control  by  a  UCS. 

•  Level  5:  Level  4  interoperability  plus  the  ability  of  the  UCS  to  launch  and  recover  the  UAV. 

To  be  interoperable  to  a  particular  level,  the  UCS  shall  be  compliant  with  all  the  requirements  stated  for  all  the 
levels  up  to  that  which  is  desired. 
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2.1.5  Legal  and  Moral  Issues 

Air  Chief  Marshall  Sir  Brian  Burridge  has  expressed  concerns  about  how  international  law  may  interpret  the 
legality  of  future  warfare  reliant  on  robotic  operations.  During  his  closing  address  to  the  ‘Iraq  2003  - 
Air  Power  Pointers  for  the  Future’  conference,  the  Air  Chief  Marshal  presented  his  views  on  the  use  of 
UCAVs  in  combat,  with  the  following  observations: 

66  When  we  go  into  combat,  we  have  got  to  be  sure  what  we  are  doing  is  both  legal  and  moral  I  do 
not  believe  that,  in  future,  even  though  technology  will  allow  it,  we  will  be  allowed  to  indulge  in 
robotic  warfare.  I  simply  do  not  see  the  international  community  regarding  that  as  an 
appropriate  way  to  fight.  The  notion  of  using  UCAVs  controlled  from  10  time  zones  away  to 
prosecute  a  battle  is  not  something  international  law  of  the  future  will  regard  as  acceptable. 

I  think  the  notion  of  a  person  in  the  loop,  the  notion  of positive  ID,  the  notion  of  someone  feeling 
the  texture  of  what  is  going  on  in  the  battlespace,  is  going  to  be  more  and  more 

prevalent . Overall,  I  think  robotic  warfare  drives  you  away  from  what  I  term  as  emotional 

connectivity  with  the  battlespace.  My  view  is  that  winning  the  hearts  and  minds  battle  with  the 
indigenous  population  requires  this  emotional  connectivity ”  [1]. 

Kennett  [2]  provides  a  detailed  analysis  of  the  legal  and  moral  issues  of  fully  autonomous  UCAV  operations. 
This  analysis  observes  that  the  rapid  technological  advances  mean  that  the  concept  of  UCAV  operating  in  a 
fully  autonomous  mode,  engaging  enemy  targets  without  human  interference,  is  nearing  reality.  It  considers 
the  implications  in  terms  of  technological  plausibility  together  with  the  legal  and  moral  aspects.  This  leads  to 
the  conclusion  that  although  technology  may  allow  waging  a  war  which  is  free  of  risk,  there  are  significant 
legal  and  moral  hurdles,  which  will  need  serious  consideration  throughout  the  development  of  such  systems. 
Consideration  of  the  technological  viability  and  legal  constraints  suggests  that  a  “human-in-the-loop”  system 
will  be  the  most  likely  mode  of  operation. 

The  analysis  considers  use  of  UCAVs  in  combat  with  focus  on  the  method  of  supervision  of  autonomous 
operations.  It  asserts  that  that  there  is  a  need  to  determine  precisely  what  that  level  of  supervision  entails  in 
conflict  and  how  best  to  ensure  that  the  human  supervisor  can  interact  efficiently  and  effectively  with  his 
uninhabited  “charges”.  The  argument  considers  that  the  possibility  of  highly  reliable  automatic  target 
recognition  (ATR)  capability  allows  the  technical  possibility  of  fully  autonomous  target  engagement. 
High  levels  of  autonomy  are  projected  for  future  UCAV  operations.  Indeed,  the  analysis  quotes  from  informed 
sources  that  it  will  be  technically  feasible  for  UCAV  to  prosecute  an  attack  fully  autonomously  with  ATR 
when  communications  breakdown  prevents  human  supervision  and  authorisation  -  but  the  question  remains, 
“what  would  happen,  for  example,  if  during  the  period  of  lost  communications  the  context  of  the  conflict 
shifted  significantly?”  Surrender,  shift  of  allegiances  of  major  actors  and  renewed  targeting  intelligence  are 
cited  examples.  It  follows  that  the  legal  requirements  for  discrimination  and  humanity  and  accountability  will 
always  require  that  a  human  authorises  the  final  decision  to  attack. 

2.1.5.1  Legal  Issues 

The  Law  of  Armed  Conflict  (LOAC)  is  a  part  of  International  Humanitarian  Law.  International  Humanitarian 
Law  is  a  combination  of  treaty  law  (binding  on  signatories)  and  customary  (binding  on  all).  Treaty  law 
requires  that  for  new  weapon  systems,  at  all  stages  of  development,  there  is  an  obligation  to  determine 
whether  its  employment  would  in  some,  or  all  circumstances  be  prohibited  by  the  protocol  or  by  any  other  rule 
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of  international  law.  All  weapons  systems  must  undergo  a  thorough  legal  review  before  entry  into  service. 
Legal  advice  is  needed  to  ensure  that  not  only  that  the  end  product  has  some  utility,  but  also  that  no 
prohibitions  exist  as  to  the  actual  development  of  certain  weapons.  Kennett  quotes  Best  [3,  p.  62]: 

“The  history  of  warfare  has  been  repeatedly  punctuated  by  allegations  that  certain  new  weapons 
are  ‘unlawful’,  because  in  some  way  unfair  by  the  prevailing  criteria  of  honour,  fairness  and  so 
on,  or  because  nastier  in  their  action  than  they  need  be” 

The  development  and  ownership  of  UCAV  platforms  is  unlikely  to  become  an  issue.  What  is  important, 
Kennett  argues,  is  that  in  the  case  of  UCAVs,  the  regulations  and  limitations  concerning  its  use  require  legal 
review.  Like  guns,  they  are  not  necessarily  illegal,  but  there  are  restrictions  on  who  can  own  them,  for  what 
purpose  and  how  they  are  employed.  The  use  of  UCAVs  in  conflict  is  similar  to  that  of  a  manned  platform. 
However,  the  major  difference  is  that 

“with  a  manned  platform  the  pilot  acts  as  additional  layer  of  authorisation  and  he  is  ultimately 
held  responsible  for  the  outcome  of  his  actions  ”. 

Thus  what  the  level  of  UCAV  supervision  entails  needs  legal  review  in  the  areas  of  discrimination,  humanity 
and  accountability. 

2. 1.5.2  Discrimination 

The  pilot  is  required  to  discriminate  between  what  is  a  valid  military  target  and  what  is  not.  Kennett  [2] 
discusses  how  information  is  interpreted  with  reference  to  the  “feel”  of  the  situation,  which  would  be  difficult 
to  replicate  in  machine  information  processing.  This  concerns  naturalistic  decision-making,  where  decisions 
are  based  on  cognitive  recognitional  processes,  involving  pattern  recognition,  implicit  and  explicit  knowledge, 
appreciation  of  meaning,  visualisation  and  judgement  skills,  acquired  previously  through  experience  with 
similar  situations. 

“ To  do  this,  the  pilot  has  a  great  deal  of  information  available  to  him,  some  of  which  can  only  be 
interpreted  by  the  feel  ’  of  the  situation,  which  is  sometimes  referred  to  as  the  sixth  sense.  Whilst 
it  would,  in  theory,  be  possible  to  harness  the  ability  for  a  computer  to  add  “feeling”  to  a 
decision  through  a  knowledge  base,  it  is  unlikely  that  such  a  system  could  be  fully  trusted  to  deal 
with  every  eventuality.  Additionally,  proving  that  such  a  system  is  robust  would  be  an 
enormously  difficult  tasK\ 

It  is  believed  that  the  legal  requirement  for  discrimination  is  one  that  will  be  vitally  important  in  the 
development  of  autonomous  UCAVs.  The  concept  of  discrimination  emerges  from  the  Additional  Protocol  to 
the  Geneva  Conventions  (Protocol  1),  which  states  that: 

Article  51(1):  “The  civilian  population  and  other  civilians  shall  enjoy  general  protection  against  dangers 
arising  from  military  operations.” 

Additionally: 

Article  51(4):  “Indiscriminate  attacks  are  prohibited.  Indiscriminate  acts  are: 

(a)  Those  which  are  not  directed  at  a  specific  military  objective; 

(b)  Those  which  employ  a  method  or  means  of  combat  which  cannot  be  directed  at  a  specific  military 
objective;  or 
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(c)  Those  which  employ  a  method  or  means  of  combat  the  effects  of  which  cannot  be  limited  as  required 
by  this  protocol;  and  consequently,  in  each  such  case,  are  of  a  nature  to  strike  military  objectives  and 
civilians  or  civilian  objects  without  distinction.” 

This  means  UCAV  development  having  to  provide  a  capability  to  carry  out  some  form  of  target  identification 
to  ensure  discrimination  between  civilian  and  military  targets.  Developments  in  ATR  seem  certain  to  provide 
a  robust  capability  against  traditional  military  targets  and  camouflage  tactics,  but  in  modern  asymmetric 
warfare,  well-organised  belligerents  ignore  the  legal  requirement  under  international  law  to  be  readily 
distinguished  from  the  civilian  population.  They  merge  with  the  civilian  population,  they  do  not  travel  in 
identifiable  military  vehicles  and  they  use  sophisticated  deception  tactics.  Also,  military  vehicles  may  be  used 
as  decoys  to  deliberately  cause  civilian  harm,  to  attract  public  opprobrium  against  indiscriminate  “aggressors” 
and  to  garner  further  public  support  for  their  belligerent  causes.  Thus,  in  modem  warfare,  it  is  very  difficult 
for  an  autonomous  machine  to  discriminate  between  civilians  and  military  targets.  Experienced  human 
judgement  is  needed  to  assess  complex  risks,  to  consider  both  the  immediate  and  broader  context,  to  judge  the 
consequences  and  implications  of  action,  and  if  possible,  to  anticipate,  see  through  and  counter  any  new 
deception  tactics.  Consequently,  any  autonomous  system  will  remain  dependent  upon  ‘human-in-the-loop’ 
targeting  decisions,  where  a  human  makes  the  ultimate  decision  to  engage  a  target. 

2. 1.5.3  Humanity 

The  concept  of  humanity,  like  discrimination,  also  emerges  from  the  Additional  Protocol  to  the  Geneva 
Conventions  (Protocol  1),  which  states  that: 

Article  35,(1):  “In  any  armed  conflict,  the  right  of  the  parties  to  the  conflict  to  choose  the  method 
or  means  of  warfare  is  not  unlimited.” 


and: 


Article  35,  (2):  “It  is  prohibited  to  employ  weapons,  projectiles  and  material  and  methods  of 
warfare  of  a  nature  to  cause  superfluous  injury  or  unnecessary  suffering.” 

The  1980  Convention  on  Certain  Conventional  Weapons  is  the  only  instrument  that  actually  prohibits  or 
restricts  the  use  of  certain  types  of  conventional  weapons.  Currently,  this  covers  non-detectable  fragments, 
mines,  incendiary  weapons  and  blinding  laser  weapons,  but  it  is  intended  to  be  capable  of  evolving  as 
technology  and  international  opinion  develop.  Kennett  suggests  that  as  autonomous  control  techniques 
develop,  their  use  probably  will  be  subjected  to  a  fair  degree  of  scrutiny  and  restriction  through  this 
convention.  It  would  be  logical,  therefore,  to  allow  for  flexibility  in  their  concept  of  use  as  designs  for 
autonomous  UCAVs  progress.  This  is  a  further  reason  why  these  systems  should  retain  the  ability  for  the 
human-in-the-loop  to  make  the  targeting  decision. 

2. 1.5.4  Accountability 

Kennett  [2]  argues  that  the  most  compelling  reason  for  maintaining  the  human-in-the-loop  targeting  decision, 
is  probably  the  requirement  for  accountability.  International  humanitarian  law  exists  so  that  those  who  fail  to 
respect  the  laws  are  open  to  legal  recrimination.  In  combat,  individuals  are  accountable  for  their  actions. 
Accountability  forces  an  operator  to  ensure  that  specific  RoE  are  met  before  the  initiation  of  any  hostile 
action.  Failure  to  adhere  to  RoE  is  likely  to  result  in  legal  action  against  that  individual. 

With  fully-autonomous  UCAVs  and  without  a  human-in-the-loop,  it  is  unclear  who  would  be  held  accountable 
should  things  go  wrong.  Kennett  asks:  “ Is  it  the  designers,  the  software  writers,  the  commander  who  tasked  the 
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mission  or  the  individual  responsible  for  “supervising”  the  autonomous  machine?  Accountability  might  not  be 
an  issue  were  very  highly  reliable  machines  to  be  possible  that  do  not  make  mistakes,  but  as  discussed 
previously  the  possibly  of  enemy  deception  can  not  be  discounted. 

The  1922  Hague  Rules  of  Air  Warfare,  which  contains  much  customary  law,  states  that  “A  military  aircraft 
shall  be  under  the  command  of  a  person  duly  commissioned  or  enlisted  in  the  military  service  of  the  State; 
the  crew  must  be  exclusively  military”.  This  rule  originally  concerns  the  use  of  civilian  crews  to  operate 
military  aircraft.  However,  the  underlying  assertion  that  “military  aircraft  must  be  under  command”  is  one  that 
could  apply  specifically  to  UCAVs.  This  has  significant  implications  for  accountability.  As  noted  earlier, 
it  will  be  technically  feasible  for  a  UCAV  to  prosecute  an  attack  fully  autonomously  with  ATR  when 
communications  breakdown  -  but  this  means  being  willing  to  trust  the  UCAV  to  determine  whether  RoE  are 
met  -  in  effect,  for  it  to  continue  its  mission  with  nobody  in  command.  This  is  to  persist  with  loss  of  human 
sensitivity  to  shifts  in  context,  and  any  resultant  possibility  of  doubt  about  engagement,  which  would  probably 
cause  the  human  not  to  engage.  Kennett  notes:  “It  is  hard  to  imagine  a  machine  that  has  doubt,  especially 
after  meeting  pre-programmed  criteria  such  as  probability  of  identification  and  particularly  when  major  shifts 
in  context  occur ”.  He  postulates  that  “the  reaction  of  the  international  legal  community  to  the  prospect  of 
autonomous  UCAVs  continuing  missions  when  communications  have  been  severed  and  thus  the  vehicle  is  not 
under  human  command,  is  likely  to  be  severe”. 


2. 1.5.5  Legal  Implications 

Kennett  [2]  indicates  that  whether  by  customary  international  humanitarian  law  or  treaty  law,  military  forces 
must  ensure  that  the  requirements  of  discrimination  and  humanity  are  met  and  that  a  degree  of  accountability 
exists.  The  following  conclusions  are  drawn  from  the  analysis: 

•  Technology  is  moving  rapidly  to  a  point  where  the  requirement  of  discrimination  could  be  met  to  a 
high  degree  of  reliability. 

•  This  technology  will  be  of  limited  utility  against  an  enemy  who  is  deliberately  willing  to  blend  into 
the  surrounding  civilian  population. 

•  The  ownership  of  UCAVs  may  be  transparent  to  the  concept  of  humanity,  but  their  use  is  likely  to 
attract  close  scrutiny  in  any  forthcoming  conflict. 

•  As  the  number  and  type  of  autonomous  combat  machines  increases,  it  is  unlikely  that  the  international 
community  will  accept  such  proliferation  without  insisting  on  new  legislation  to  limit  or  restrict  their 
use. 

•  Accountability  will  remain  one  of  the  key  tenets  of  future  conflict,  and  whilst  autonomous  combat 
machines  promise  much,  it  is  likely  that  humans  will  always  be  required  to  make  decisions  where 
humans  may  perish. 

2.1.5.6  Moral  and  Ethical  Issues 

The  study  of  moral  issues  (ethics)  is  “concerned  with  or  relating  to  the  distinction  between  good  and  bad  or 
right  and  wrong  behaviour ”  (based  on  the  definition  of  ‘moral’  from  the  Oxford  English  Dictionary). 
In  considering  moral  issues  of  using  autonomous  UCAVs,  Air  Chief  Marshall  Sir  Brian  Burridge  refers  to  the 
term  “morality  of  altitude ”  that  was  coined  in  to  reference  the  disconnection  of  the  pilot  at  10,000  feet  from 
the  destruction  caused  by  bombing  on  the  ground  [4].  This  disconnection  led  to  a  lower  incidence  of 
psychological  problems  amongst  USAF  pilots  than  their  US  Army  colleagues  on  the  ground  during  the 
Vietnam  conflict.  He  believes  that  the  “morality  of  altitude ”  is  at  the  heart  of  the  debate  of  how  international 
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law  will  interpret  robotic  warfare  in  the  future.  He  concludes  that  “Feeling  the  granularity  of  the  battle-space 
is  the  key  issue  in  interpreting  the  Rules  of  Engagement ’. 

The  Air  Chief  Marshall  poses  the  future  possibility  of  the  “Play  Station”  operator  who  may  never  have  had 
actual  combat  experience,  no  connections  with  other  operational  units,  and  no  shared  operational  experiences 
[4].  Furthermore,  he  expects  that  future  highly  autonomous  systems  will  be  reliant  on  an  experienced 
programmer  for  their  autonomy,  who  may  not  have  any  experience  of  combat  operations  in  a  manned 
platform.  He  notes  that  this  will  “further  remove  the  remote  pilot  from  the  system  and  place  him  within  the 
industrial  or  military  support  base ”.  The  Air  Chief  Marshall  discusses  how  this  exacerbates 
“the  disconnection  of  air  power  from  the  shared  battle-space’'.  In  considering  the  increasing  lethality  and 
persistence  of  UAVs,  he  questions  how  we  stop  the  “Play  Station ”  generation  becoming  the  “playground 
bully ”  of  the  battlefield?  He  asks  if  this  disconnection  exacerbates  the  potential  for  the  “play  ground  bully ”  in 
all  of  us  to  emerge.  He  contrasts  the  simplicity  of  “drop  and  drag ”  mouse  actions  on  a  lap  top  during  remote 
reach-back  operations,  with  the  consequences  on  the  other  side  of  the  world. 

This  discourse  has  some  resonance  with  the  social  psychology  of  obedience  and  capability  for  human  to  act 
callously  when  disconnected  from  suffering  and  under  pressure  from  authorities  to  inflict  harm  [5,6].  Milgram 
famously  created  a  memory  and  learning  experimental  situation  in  which  volunteer  subjects  believed  they  were 
administering  electric  shocks  of  increasing  severity  to  another  individual,  as  punishments  for  mistakes  in  two 
word  pair  reading  lists.  No  shocks  were  actually  administered,  but  the  subjects  were  made  aware  of  the 
discomfort  caused  by  poundings  on  the  wall.  Quoting  Milgram:  “With  numbing  regularity  good  people  were 

seen  to  knuckle  under  the  demands  of  authority  and  to  perform  actions  that  were  callous  and  severe . 

A  substantial  portion  of  the  people  do  what  they  are  told  to  do,  irrespective  of  the  content  of  the  act  and  without 
limitations  of  conscience,  so  long  as  they  perceive  that  the  command  comes  from  a  legitimate  authority ”. 

Drop  and  drag  mouse  actions  at  remote  UAV  control  stations  seem  even  more  dissociated  from  their  effects 
than  Milgram’ s  electrical  switches.  Unless  the  “Play  Station ”  generation  can  somehow  avoid  “the  morality  of 
altitude ”  and  “feel  the  granularity  of  the  battle-space ”,  we  probably  should  not  be  surprised  if  they  have  the 
potential  to  become  the  “playground  bully ”  of  the  future  battlefield,  and  find  UMV  operations  running  into 
conflict  with  international  law. 

Kennett  [2]  reviews  the  argument  for  the  morality  of  using  autonomous  UCAVs  to  kill  without  risk. 
This  analysis  concludes  that,  in  some  circumstances,  even  though  technology  and  the  law  will  allow  such  acts, 
morally,  it  would  be  indefensible.  The  analysis  begins  by  recognising  that  killing  in  conflict  presents  a  moral 
dilemma  and  that  there  is  no  universally  accepted  view  of  the  truth.  Feeling  or  subjectivity,  as  opposed  to 
reason,  largely  determines  the  moral  justifications  -  whether  it  feels  truly  right  or  wrong.  The  ethical 
implications  will  be  dependent  on  the  situation,  ranging  from  supreme  emergency  to  war  of  choice.  In  a  war 
of  choice,  what  is  morally  acceptable  behaviour  will  be  comparatively  restrictive  and  judged  on  a  sliding  scale 
of  necessity  in  the  socio-political  rather  than  legal  sense.  Using  UCAVs  in  a  war  of  national  survival  will  be 
morally  acceptable.  During  a  war  of  choice,  pitted  against  a  technologically  inferior  opponent,  using  UCAVs 
will  be  acceptable  for  attacking  targets  that  represent  either  an  imminent  or  clearly  identifiable  latent  threat 
(ballistic  missiles,  WMDs,  massed  armoured  forces)  -  but  although  legally  justifiable,  it  would  be  more 
difficult  to  feel  justified  attacking  small  groups  of  military  personnel  with  UCAVs,  decided  by  operators  who, 
as  Burridge  [4]  describes,  are  “10  time  zones ”  away,  removed  from  the  threat,  free  of  risk  and  without 
“the  emotional  connectivity  of  the  battlespace” .  Kennett  adds  “to  open  fire  and  to  take  life  in  such  a  situation 
does  seem  somewhat  unsporting,  even  in  warfare ”.  What  emerges  is  that  the  taking  of  life,  at  a  time  and  place 
of  ones  own  choosing  entirely  without  risk,  although  legally  acceptable  in  war,  is  morally  unjustified.  There 
has  to  be  an  emotional  connectivity  with  the  decision,  arising  from  having  to  endure  feelings  of  risk,  sufficient 
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to  consider  whether  the  decision  feels  right...  “Having  to  endure  risk  indicates  no  choice  but  to  be 
emotionally  connected  to  the  battlespace” .  It  does  not  sound  right  to  be  able  to  wage  war,  risk  free,  through 
the  virtue  of  technology,  and  in  the  course  of  doing  so,  to  kill  regularly  so  that  killing  becomes  a  de-personalised 
act.  The  analysis  concludes: 

“There  will  always  be  times  when  killing  without  risk  in  conflict  will  be  necessary  and  entirely 
justifiable.  However,  outside  the  boundaries  of  fighting  for  national  survival,  such  acts  should 
not  become  the  norm.  To  allow  this  would  be  morally  wrong  and  risks  devaluing  the  serious 
nature  of  conflict  through  a  lowering  of  the  moral  value  of  killing  and  the  value  of  human  life ”. 
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2.1.6  UMV  Use  Cases 

An  articulation  of  the  context  of  use  of  UMV  systems  is  needed  in  order  to  provide  a  military  relevant 
reference  basis  for  considering  HF  issues.  Use  cases  can  be  derived  from  existing  representative  military 
scenarios  and  associated  66 snapshots ”  or  66 vignettes ”.  Military  scenarios  are  ways  of  characterising  future 
military  threats  and  challenges.  Scenarios  are  tools  for  operations  analysis.  They  provide  a  means  for 
evaluating  capabilities  and  capability  gaps,  investigating  military  effectiveness  and  estimating  cost/benefits 
for  investment  appraisals.  In  addition,  scenarios  are  tools  for  systems  engineering,  providing  the  basis  for  the 
mission  analyses  used  to  develop  understanding  of  system  functions,  information  and  user  interface 
HF  requirements.  The  selection  of  vignettes  is  important  and  entails  the  inherent  risks  of  being 
unrepresentative.  Selectivity  risks  biasing  analysis,  by  introducing  irrelevant  or  limiting  focus,  and  setting 
overly  restrictive  boundaries  and  limitations  to  thinking.  On  the  other  hand,  representative  use  cases,  obtained 
from  credible  and  knowledgeable  sources,  provide  an  efficient  and  effective  means  of  communicating 
requirements  and  sharing  understanding  on  likely  contexts  of  use.  This  is  particularly  valuable  for  considering 
the  relevance  of  new  concepts  and  technologies.  Development  of  system  engineering  requirements  for  UMVs 
is  beyond  the  scope  of  the  work  of  HFM-018.  The  purpose  of  this  report  is  to  provide  a  basis  for  discussion  of 
scientific  research  on  UMV  technology  and  associated  HF  issues.  Use  cases  are  provided  here  to 
communicate  likely  contexts  of  use  for  appraisal  of  the  military  relevance  of  scientific  and  technical  content 
of  the  report  in  the  chapters  that  follow. 
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2. 1.6.1  UAV  Use  Case 

Military  Applications  Study  NATO  SAS-016,  Future  Operations  with  UAV  Systems  and  SCI- 124  on 
UAV  C2  Architectures,  have  developed  a  Peace  Support  Operations  (PSO)  scenario  to  describe  understanding 
UAV  roles  [1].  The  PSO  scenario  is  considered  to  be  suitable  reference  use  case  for  the  air  component  of 
HFM-018  work.  It  is  derived  from  a  high  level  Peace  Enforcement  scenario  and  set  during  an  early  phase  of 
the  overall  NATO  operation.  NATO  forces  conduct  major  air  operations  (offensive,  defensive,  support)  using 
Composite  Air  Operations  (COMAO)  conducted  from  a  Combined  Air  Operations  Centre  (CAOC). 
The  COMAO  aims  to  maximise  the  impact  of  a  given  force  by  concentrating  efforts  and  enhancing 
survivability.  This  is  achieved  through  mutual  support,  best  available  use  of  assets  and  saturation  of  air 
defences.  The  challenges  are  primarily  from  integrated  air  defences.  This  phase  of  the  COMAO  is  illustrated 
in  Figure  2-2  below.  Figure  2-2  depicts  the  initial  forced  entry  of  NATO  amphibious  and  airborne  forces  into 
the  crisis  area,  the  establishment  of  safe  deployment  areas  and  facilities,  and  the  deployment  of  lead  elements 
of  the  main  PSO  force  and  supplies  into  those  areas. 


Figure  2-2:  Peace  Enforcement  -  Peace  Support  Operations  at  COMAO  Level. 


During  this  phase  NATO  forces  seize  and  hold  seaports  and  airports  to  be  used  for  entry.  At  the  end  of  this 
phase  all  entry  forces  have  been  deployed  into  the  crisis  country  and  the  entry  zones/bridgeheads  are  secure. 
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The  operational  scenario  covers  a  limited  period  of  time  (10  days).  During  this  time  the  various  operations  and 
missions  take  place.  Although  it  is  derived  from  the  high  level  scenario,  certain  aspects  of  the  operational 
scenario  have  been  generalised  in  order  to  avoid  making  it  too  geographically  and  threat  specific.  At  the  time 
at  which  this  ‘snapshot’  operational  scenario  is  set  the  three  main  landing  areas  have  been  established  and 
NATO  forces  have  advanced  up  to  25  km  from  the  entry  zones. 

The  over-arching  objectives  which  NATO  air  power  must  accomplish  and  the  tasks  and  functions  it  must 
perform  in  the  operational  scenario  are  as  follows: 

•  Neutralisation  of  the  internal  air  threat  (No  Fly  Zone); 

•  Defensive  counter  air  operations; 

•  Offensive  counter  air  operations  against  air  bases; 

•  Neutralisation  of  ground-based  threats  to  air  traffic; 

•  Radar  support  j amming; 

•  Hard  kill’ s/DEAD; 

•  Communications  j  amming  support; 

•  Offensive  Air  Support  for  NATO  entry  forces  and  lead  elements  of  the  main  PSO  force; 

•  Deep  interdiction  of  opposition  ground  combat  power; 

•  Neutralisation  of  the  maritime  threat; 

•  Battlespace  surveillance,  airborne  command  and  control  and  communications  support; 

•  Air  surveillance; 

•  Ground  surveillance  and  reconnaissance; 

•  Maritime  surveillance  and  reconnaissance; 

•  Battlefield  tactical  command  and  control; 

•  Re-supply  of  NATO  forces; 

•  Emergency  non-combatant  evacuation; 

•  Combat  Search  and  Rescue;  and 

•  Psychological  operations. 

NATO  forces  must  also  deter  intervention  and  attacks  by  states  neighbouring  the  crisis  country.  As  such, 
NATO  air  power  is  also  held  in  readiness  to  deal  with  air  and  ballistic  missile  attacks  from  these  neighbours, 
to  interdict  any  ground  attack,  and  to  conduct  operations  for  strategic  effect  against  them.  However,  at  the 
time  at  which  the  operational  scenario  is  set  no  attacks  have  been  launched  by  these  neighbouring  states. 
Consequently  all  NATO  air  activity  in  the  snapshot  operational  scenario  is  related  to  the  main  Peace 
Enforcement  operation  in  the  crisis  country. 

The  air  defence  and  offensive  air  capabilities  of  the  opposition  forces  have  been  substantially  degraded  by  the 
time  of  this  operational  scenario.  However,  the  opposition  air  defences  have  not  attempted  to  mount  a  full- 
scale  challenge  to  NATO  offensive  air  operations.  Similarly,  the  opposition  has  conducted  only  limited 
offensive  air  operations.  Rather  the  aim  of  opposition  air  defence  and  offensive  air  operations  has  been  to 
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inflict  losses  on  NATO  forces  that  although  of  limited  military  operational  and  tactical  significance, 
the  opposition  believes  will  have  direct  strategic  and  political  impact. 

As  a  consequence  the  opposition  has  sought  to  preserve  his  air  defence  and  offensive  air  capabilities  by  avoiding 
large-scale  engagements,  and  by  trying  to  make  his  forces  difficult  to  locate  and  target.  Consequently,  at  the  time 
at  which  this  operational  is  set,  the  opposition  forces  still  have  some  air  defence  and  limited  offensive  air 
capabilities. 

2. 1.6.2  Integration  of  UAV  and  Manned  Aircraft  Systems 

UAV  currently  operate  in  segregated  airspace  physically  separated  from  manned  aircraft.  SCI- 124  report  that 
this  is  due  to  the  lack  of  clearance  to  operate  in  conjunction  with  manned  aircraft  and  other  UAVs,  and  overall 
inexperience  with  UAV  operations.  Integration  with  manned  aircraft  in  the  same  airspace  will  soon  be  enabled 
by  technological  advances.  Currently,  the  most  common  role  for  UAVs  is  ISR.  UAVs  are  performing  limited 
strike  missions.  Dedicated  strike  UAV  platforms  are  planned  for  the  mid-term  timeframe.  Therefore,  SCI- 124 
have  concluded  that  there  is  a  definite  requirement  for  the  integration  of  strike  UAVs  with  manned  aircraft. 

NATO  SAS-16  used  analysis  of  the  PSO  scenario  and  UAV  capabilities  to  derive  conclusions  on  UAV 
integration.  An  illustration  of  the  integration  of  UAVs  into  the  PSO  scenario  is  shown  in  Figure  2-3.  Different 
classes  of  UAV  systems  can  undertake  different  types  of  mission/tasks  dependent  on  payload  (mass,  type), 
range  and  endurance,  speed  and  survivability  (altitude,  speed,  signature,  defensive  aids).  Broadly,  they 
conclude  that  in  the  mid  term  UAVs  will  be  able  support  only  a  limited  range  of  attack  missions.  In  long  term, 
a  wider  range  of  attack  missions  could  be  supported,  with  greater  autonomous  operations,  and  tight  rules  of 
engagement,  involving  human-in-the-loop  during  target  engagement  phases.  At  the  mission  level,  concepts  of 
employment  will  depend  heavily  on  mission  and  UAV  system  type.  UAV  systems  can  replace  manned  aircraft 
for  various  mission/tasks,  placing  fewer  aircrew  at  risk.  UAV  systems  can  complement  manned  aircraft  to 
enhance  effectiveness  of  manned  attack  missions.  They  can  perform  high  risk,  high  pay-off  missions/tasks. 
Also,  long  endurance  UAV  systems  can  support  more  than  one  mission. 
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NATO  SAS-16  concluded  that  many  concepts  require  close  co-ordination  between  the  UCS  and  manned 
attack  aircraft,  but  close  co-operation  is  already  needed  on  manned  aircraft  missions  [1].  Some  concepts 
involve  hand-over  of  control  to  manned  aircraft.  Where  hand  over  occurs  multi-crew  types  offer  a  better 
option,  and  there  is  a  risk  of  overloading  the  mission  crew  on  airborne  C2  platforms.  Data  links  to  the  UCSs 
are  a  key  issue  and  weakness  (jamming).  Beyond-line-of-sight  requirements  and  latency  problems  add 
complexity,  when  detailed  control  is  required.  C2  authorities  will  need  assistance  to  maintain  the  air  picture. 
For  the  mid-term,  there  will  be  no  close  formation  flying  because  of  technical  limitations  and  aircrew  mistrust. 
Special  corridors  will  be  needed  for  ingress  and  egress,  with  rendezvous  with  manned  aircraft  in  pre-planned 
areas.  Special  operating  areas  will  need  to  be  identified.  Longer  term,  there  will  be  greater  formation 
coordination  and  integration  with  manned  aircraft,  since  manned  aircraft  will  remain  in  service  for  many 
years,  and  integration  with  the  C4ISR  system. 

For  COMAO  mid  term,  SAS-016  observed  that  UAVs  can  replace  various  manned  aircraft  missions,  but  not 
those  involving  close  formation  coordination.  UAV  systems  will  use  assigned  corridors  (routes  and  flight 
levels).  UCSs  will  need  to  be  integrated  into  the  COMAO  planning  process.  This  will  include  consideration 
of:  assignment  of  UAV  mission  leaders;  routing  (rendezvous  areas  and  procedures,  timings,  air  traffic 
procedures);  reaction  to  threats;  target  area  tactics  and  procedures;  delay  and  cancellation  procedures  and 
options;  communications  and  data  link  selection  and  de-confliction,  including  beyond-line-of-sight 
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requirements;  UAV  handover  procedures  where  appropriate.  The  potentially  large  number  of  UCSs  will  need 
rationalisation  and  standardisation  of  the  ground  segment  of  UAV  systems.  UCS  standardisation  has  been 
undertaken  through  STAN  AG  4586  activities  [2].  Current  COMAOs  with  manned  aircraft  require  extensive 
pre-planning  and  have  a  degree  of  rigidity.  The  introduction  of  UAVs  will  shift  the  workload,  but  should  not 
necessarily  lead  to  overall  increases.  Longer  term,  UAV  systems  for  attack  missions  (UCAV)  will  not  need 
support,  and  there  will  be  reduced  demand  for  COMAO.  Surveillance  and  reconnaissance  UAV  systems  will 
form  an  integrated  part  of  the  overall  C4ISR  system.  Direct  links  to  offensive  air  operations  will  reduce  the 
decision  loop  times  of  the  sensor-to-shooter  system.  At  the  operations  level,  UAV  system  planning  will  need 
to  be  incorporated  at  the  CAOC/Air  operations  level  (Air  Tasking  Order  -  ATO;  Airspace  Control  Order  - 
ACO;  Airspace  Control  Measures  -  ACM)).  As  a  cautionary  observation,  SAS-016  noted  that  UAV  systems 
may  lend  themselves  to  micro-management  from  higher  command  levels. 

HFM-018  work  on  a  human-centric  framework  for  control  of  UAV  operations  required  a  degree  of  more 
detailed  task  specificity  than  provided  in  the  PSO  use  case.  The  following  UAV  tasks  were  identified  for  a 
COMAO  mission  to  attack  an  airfield  in  Article  V  operations,  derived  from  SCI- 124  analyses  [1]. 

Ingress 

•  Control,  Guidance,  Navigation  (ownship  and  attack  aircraft); 

•  Replan; 

•  Communication  (C2  MC,  attack  aircraft,  other  UAV  UCS); 

•  System  management  (+contingencies); 

•  Self  defence;  and 

•  Target  location. 

Over  Target 

•  Target  registration,  identification,  verification,  designation  (for  attack  aircraft); 

•  Control,  Guidance,  Navigation; 

•  System  management  ^contingencies); 

•  Communication  (C2  MC,  attack  aircraft,  other  UAV  UCS); 

•  Sensor  management; 

•  Self  defence; 

•  Rules  of  engagement;  and 

•  Battle  damage  assessment. 


Egress 

•  Control,  Guidance,  Navigation; 

•  Communication  (C2  MC,  attack  aircraft,  other  UAV  UCS); 

•  System  management  ^contingencies);  and 

•  Self  defence. 
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2. 1.6.3  UUV  and  UGV  Use  Cases 

For  the  purposes  of  this  report,  the  VSW  MCM  role  provides  a  commonly  recognised  use  case  for 
understanding  the  application  of  UUVs.  The  basic  task  structure  for  MCM  can  be  re-used  to  apply  to  UGVs  in 
the  MCM/EOD  role. 

Blackburn  et  al  [3]  describe  the  UUV  VSW  MCM  (and  EOD)  task  as  comprising  seven  phases: 

1)  Deployment  and  distribution  of  assets; 

2)  Execution  of  a  search  strategy; 

3)  Detection  of  mine-like  objects; 

4)  Classification  and  identification; 

5)  Neutralisation; 

6)  Verification  and  certification  of  clearance;  and 

7)  Recovery  of  assets. 

The  UGV  use  case  covers  MCM  and  EOD  tasks  from  high  tide  mark  and  above,  providing  route 
reconnaissance  and  access  in  support  of  troop  advancement  in  the  PSO  scenario.  The  following  tasks  can  be 
considered  to  apply  to  UGVs: 

•  Threat  Assessment  -  close  quarter/booby  trap  and  hostile  environments; 

•  Route  Opening  -  approach  dependent  on  collateral  damage  and  speed  of  advancement; 

•  Route  Clearance  -  approach  dependent  on  collateral  damage  and  speed  of  advancement;  and 

•  Casualty  recovery  and  management. 

Understanding  of  how  both  UUVs  and  Unmanned  Surface  Vehicles  (US  Vs)  could  conduct  MCM  is  described 
in  Carver  [4].  This  analysis  reports  that  the  foremost  reason  for  preferring  the  UUV  to  a  USV  in  the  MCM 
role,  is  the  requirement  to  conduct  the  mission  initially  in  a  covert  manner.  In  many  scenarios,  it  is  not  likely 
to  be  politically  acceptable  to  conduct  overt  operations  in  another  nation’s  littoral  waters,  thus  making  a  semi- 
submersible  UUV  equally  unacceptable.  In  any  scenario,  a  variety  of  tactical  options  emerge,  dependent 
largely  on  the  level  of  autonomy  given  to  the  vehicle  and  the  ultimate  mission  objectives.  In  the  MCM  role, 
the  UUV  will  be  deployed  from  a  platform,  or  a  harbour,  and  transit  to  its  operational  area.  Characteristics  of 
the  vehicle  associated  with  this  phase  include  its  range,  speed  and  endurance,  which  must  match  the  expected 
operational  mission  and  tidal  conditions.  UUV  capability  is  a  function  of  size.  The  greater  the  capability  in 
each  area  the  larger  the  vehicle  will  be,  unless  new  technology  provides  batteries  or  fuel  cells  with  a  greater 
power  density.  Once  in  theatre,  the  UUV  will  run  along  its  pre-programmed  survey  path  towards  the 
objective.  Dependent  upon  the  vehicle’s  payload  capability,  degree  of  autonomy  and  rules  of  engagement,  the 
UUV  might  then  follow  one  of  the  following  tactical  options: 

•  Report  the  position  of  mines  and  wait  for  further  instructions. 

•  Autonomously  decide  no  clear  path  through  the  minefield  exists  and  search  for  an  alternate  route  to 
the  objective. 

•  Dispose  of  mines  along  the  swept  path  (from  here  the  covert  nature  of  the  mission  is  lost). 
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The  required  operational  role  will  determine  the  payload,  which  will  influence  the  vehicle’s  size  and  the 
power  required  to  complete  the  mission.  In  all  events,  the  vehicle  will  be  fitted  with  an  inertial  navigation 
system,  collision  avoidance  sonar  and  high-resolution  mine  hunting  sonar.  Further,  net  enabled  capability 
(NEC)  will  be  provided  by  secure  AComms.  Communications  may  include  the  deployment  of  buoys  or 
returning  to  the  surface  to  communicate  directly.  The  greater  the  autonomy  given  to  the  vehicle  and  the  use  of 
buoys  will  help  to  maintain  the  covert  nature  of  the  mission.  Ordnance  will  need  to  be  carried  if  the  detected 
mines  are  to  be  disposed  of. 

Various  UUV  scenarios  appear  in  the  open  literature.  Carver  [4]  describes  how,  in  the  “harbour  mine  hunt” 
scenario,  a  number  of  small  UUVs  are  launched  to  search  for  mines.  Because  total  coverage  is  so  important, 
the  small  UUVs  would  continually  verify  their  position  using  GPS  information  provided  by  a  larger 
communications/navigation  vehicle  positioned  just  outside  the  search  area.  Once  the  mines  have  been  located, 
each  small  UUV  (e.g.,  ALUV)  could  position  itself  over  a  mine  and,  after  receiving  the  command,  drop  a 
small  detonation  charge  to  neutralise  the  mine. 

2.1.6.4  Composite  UMV  Use  Case 

For  the  purposes  of  this  report,  HFM-018  (led  by  Lt  Cdr  A.G.  Carver)  have  developed  a  composite  scenario 
snapshot  or  “vignette”,  to  provide  a  UMV  use  case  to  illustrate  the  possible  functioning  of  UMVs. 
This  composite  vignette  includes  space,  air,  sea  and  land  domains  with  UGV,  UUV  and  UAV  task  elements. 
The  tasks  are  summarised  in  Table  2-3.  The  task  links  are  illustrated  in  Figure  2-4. 
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Table  2-3:  Composite  Scenario  Tasks 


Phases 

Domains 

Space 

Air 

Sea 

Land 

Period  of 
Increased 
Tension 

Surveillance  of 
potentially  hostile 
nation  (PHN)  - 
satellite  to  look  for 
indications  of 
hostile  intent. 

Observe  forces 
infrastructure  - 
deploy  satellites 
to  observe  PHN 
force  dispositions. 

Observe  forces 
manoeuvres  -  satellite 
to  conduct  survey 
of  PHN  forces. 

Observe  potential 
landing  areas  -  use 

UAV  to  conduct  initial 
survey  of  potential 
landing  sites. 

Observe  force  numbers 
and  manoeuvres  - 
UAV  to  conduct 
survey  of  PHN  forces. 

Rapid  environmental 
assessment  -  UUV  to 
make  assessments  of 
ship  movements  and 
collect  environmental 
data. 

Transition 
to  War 

Targeting  -  satellite  to 
collect  targeting  data. 

Update  observations. 

Targeting  -  UAV  to 
collect  targeting  data. 

Force  observation. 

Maritime  survey  - 
UUVs  deployed  to 
conduct  more  detailed 
survey  of  possible 
landing  areas/locations. 
This  could  be  a  beach  or 
a  port  if  not  heavily 
defended. 

Mine  survey  -  UUV 
performs  route  surveys 
with  synthetic  aperture 
sonar.  Identify  mine  like 
objects  in  path  of  route 
to  selected  beach  head. 

Beach  survey  -  UUV 
deploys  UGV  to  gather 
initial  beach  survey  data 
ahead  of  special  forces. 

Port  survey  -  UUV 
enters  port  and  conducts 
visual  survey  of 
facilities  and  force 
disposition. 

Targeting  -  UGV  to 
collect  targeting  data. 
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Phases 

Domains 

Space 

Air 

Sea 

Land 

Conflict 

Deploy  air  to  ground 
weapons  to  destroy 
strategic  and  tactical 
targets  -  UAVs  to 
carry  offensive 
weapons  into  theatre. 

Mine  clearance  - 

1)  UUV  to  continue 
mine  survey. 

2)  UUV  to  conduct 
mine  disposal. 

3)  UUV  to  provide 
verification  and 
certification  of 
clearance. 

Support  beach  landing  - 
Large  UUV  to  transport 
equipment  and  re-supply 
special  forces  ashore. 

Route  reconnaissance 
for  access  -  UGV 
deployed  to  perform 
route  surveys  above 
water  line  in  path  of 
route  from  beach 
landing  area  ahead  of 
special  forces. 

Threat  assessment  - 
close  quarter  booby 
trap  and  hostile 
environments  -  UGV 
to  identify  mine  like 
objects,  EOD,  and 

CBRN  hazards  in 
path  of  selected  route. 

Route  clearance  -  close 
quarter  booby  trap  and 
hostile  environments  - 

1) UGV  to  conduct 
mine  disposal  and 
neutralise  EOD. 

2)  UGV  to  provide 
verification  and 
certification  of 
mine  clearance. 

3)  UGV  to  provide 
detection  and  analysis 
of  chemical,  nuclear 
and  biological 
hazards. 

Casualty  recovery  and 
management  -  UGV 
deployed  to  conduct 
CSAR,  and  casualty 
evacuation. 

Covert  operations  - 
UGV  deployed  to 
support  of  covert 
operations.  Sense  and 
report  a  personnel  and 
vehicle  presence  night 
and  day.  Report  target 
and  platform  locations. 
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Figure  2-4:  Illustration  of  Composite  Scenario  Task  Links. 
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2.2  PART  II  -  MILITARY  RELEVANCE  FOR  UNINHABITED  AERIAL 
VEHICLES  (UAVS) 

2.2.1  Introduction 

Technology  is  a  great  source  of  power,  which  also  is  the  reason  why  the  military  righteously  is  very  concerned 
with  it.  New  techniques  create  new  tools  for  the  human  to  utilise,  new  ways  for  the  military  to  impose  force. 
Together  with  the  development  of  the  logical  circuit  follows  however  a  significant  increase  of  the  rate  by 
which  technological  systems’  ability  to  do  new  things  increases.  Systems  are  quickly  getting  performance 
characteristics  that  not  only  depend  on  concrete  physical  input,  but  also  on  arbitrary  and  abstract  contexts. 
With  the  help  of  the  microprocessor  and  clever  programming  skills,  technology  has  expanded  from  advanced 
mechanics  to  automated  machines  that  use  logic  and  artificial  intelligence.  Technology  has  with  that  taken  the 
great  leap  into  an  area  that  used  to  be  strictly  human  business,  the  world  of  abstracts. 

This  development  is  likely  to  be  just  like  any  other  development,  humanity  will  eventually  leam  how  to  best 
use  it  and  certainly  benefit  from  it  -  but  there  will  always  emerge  a  few  solutions  along  the  way  that  probably 
would  have  been  better  off  never  being  made. 

There  is  however  a  certain  peculiarity  about  abstracts:  everything  is  more  or  less  interconnected.  It’s  quite 
impossible  to  state  where  one  issue  ends  and  the  other  starts.  This  means  that  almost  every  tiny  little  abstract 
rule  somehow  ends  up  being  dependent  on  the  supreme  abstract  dilemmas  as  aim,  purpose  and  eventually 
even  ethics  and  morale.  Hence,  it’s  not  as  easy  to  let  technology  perform  only  delimited  parts  of  the  abstract 
work  as  is  possible  with  physical. 

Technology  will  for  certain  be  of  great  assistance  in  the  traditionally  strictly  human  area  of  abstracts,  but  the 
rules  are  different.  This  text  argues  that  abstract  tasks  performed  by  technology  must  be  designed  with  a 
greater  amount  of  co-operation  between  human  and  machine,  compared  with  typical  physical  work. 
Automated  tasks  depending  on  abstract  contexts  must  be  designed  with  great  care.  The  more  abstractly 
complicated  a  machine  becomes  the  more  thought  must  be  put  behind  the  design  of  the  human  role  in  order  to 
create  a  machine  that  will  be  truly  useful,  even  if  the  overall  context  differs  from  first  assumptions. 

The  main  issue  for  this  text  is  that  such  thinking  is  believed  to  require  a  minor  shift  in  focus  when  designing 
systems:  it  requires  a  human-oriented  philosophy.  Although  the  tone  in  the  philosophy  presented  here  might 
indicate  that  certain  things  are  impossible,  that’s  not  the  tenor,  this  is  only  to  bring  matters  to  a  head. 
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Scope:  The  arguments  need  to  be  viewed  differently  for  every  context.  Human  control  of  a  tiny  little  device 
working  alone  does  in  fact  embrace  these  issues,  although  only  very  philosophically  -  but  when  it  comes  to 
complicated  systems  affecting  really  serious  matters  then  this  philosophy  should  have  significant  implications 
on  just  about  everything  from  design  to  application.  When,  for  instance,  it’s  stated  that  only  humans  are 
allowed  to  judge  in  the  topmost  context  and  therefore  need  to  be  in  control,  it  doesn’t  say  that  humans  need  to 
do  everything.  The  human  may  justifiably  have  countless  more  or  less  completely  automatic  systems  doing 
lots  of  good  work,  but  this  philosophy  emphasises  that  there  is  an  important  question  about  how  these  systems 
affect  the  ability  for  the  human  to  make  sensible  decisions  and  influence  the  situation. 

From  the  Swedish  UCAV  study’s  point  of  view  this  material  serves  as  one  input  among  others  to  the 
compilation  of  our  theoretical  framework.  Its  purpose  is  to  be  a  believed  necessary  counterbalance  to  the 
perceived  technologically  focused  UAV  community.  It  should  not  be  seen  as  an  official  standpoint. 

2.2.2  The  Map  of  Relevance 

The  entrance  of  technology  into  the  area  of  abstracts  has  introduced  the  possibility  to  develop  automation, 
which  in  turn  has  made  it  possible  to  design  uninhabited  platforms.  These  technologies  provide  a  lot  of  new 
exciting  opportunities,  not  least  for  many  typical  military  situations,  which  tend  to  be  extremes  both  regarding 
performance  and  risk.  There  is  however  a  significant  risk  of  counterproductive  solutions,  regarding  actual 
effect  in  reality.  The  relative  strengths  of  automation  and  uninhabited  vehicles  are  quite  direct  results  of 
platform,  vehicle  or  system  characteristics  and  thus  are  comparatively  easy  to  spot.  The  relative  weaknesses 
on  the  other  hand,  are  mostly  more  indirect  consequences  of  loss  of  human  control  as  consequence  of 
conditions  created  by  the  characteristics  of  the  systems.  These  consequential  weaknesses  predominantly 
become  apparent  in  situations  with  significant  uncertainties,  especially  when  there  are  uncertainties  in 
arbitrary  abstract  dimensions  including  context  and  politics,  in  situations  with  great  complexity  and 
sensitivity.  They  tend  to  show  up  in  actual  conflict  situations  where  “reality  suddenly  comes  and  hit  you  in  the 
face”.  Also,  the  only  fixed  planning  parameter  for  future  conflicts  is  that  there  is  significant  uncertainty  about 
what  the  situations  actually  will  look  like. 

To  appropriately  judge  the  relevance  of  anything,  both  strengths  and  weaknesses  must  be  known  and  assessed 
according  to  a  common  ground  of  values.  For  something  to  be  military  relevant  there  must  be  a  desired 
military  effect  that  exceeds  the  cost  of  using  it.  And  military  effect  is  nothing  but  humans  influencing  other 
humans  using  military  powers,  commonly  facilitated  by  technological  systems. 

The  Map  of  Relevance  consists  of  a  collection  of  theoretical  descriptions  of  a  few  believed  fundamental 
conditions  in  [military]  reality  including  an  attempt  to  describe  the  subtle  but  important  difference  between 
designed  and  applied  [military]  effect.  Appropriate  effect  is  the  key  to  relevance  and  this  philosophy  argues 
that  the  only  relevant  [military]  effect  is  such  that  is  intentionally  achieved  by  [military]  humans,  where 
intentionally  meaning  being  completely  in  charge  of  what’s  happening.  If  not,  then  the  effect  isn’t  really 
military  and  thus  has  no  military  relevance.  That  is,  a  human  in  charge  is  unquestionable  and  hence  the  reason 
to  state  it  as  an  axiom. 

2.2.3  The  Human  Axiom 

The  purpose  of  the  human  axiom  is  not  to  oppose  automation  and  uninhabited  systems  since  these 
technologies  do  render  important  capabilities.  The  purpose  is  to  provide  a  foundation  for  the  believed 
necessary  thinking  required  for  developing  better  systems  containing  automation.  It’s  based  on  a  human- 
oriented  philosophy  emerged  from  the  issue  of  uncertainty  and  the  theories  of  control  described  below. 
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2.2.3. 1  The  Philosophy 

This  philosophy  may  be  interpreted  somewhat  controversial  and  is  just  because  of  that  believed  to  be  relevant. 
It’s  intentionally  a  bit  provocative  due  to  the  ambition  of  being  an  eye-opener  and  needs  to  be  read  with  that  in 
mind.  Because,  it’s  not  fundamentally  important  whether  the  word  is  autonomous  or  highly  automated, 
whether  mankind  suffers  from  ugly  technological  hysteria  or  simply  is  a  bit  technologically  focused,  but  the 
philosophy’s  underlying  “crux  of  the  matter”,  the  essence  that  perhaps  isn’t  really  managed  to  be  clearly 
conveyed,  but  that  hopefully  at  least  is  written  somewhere  between  the  lines,  is  still  very  much  the  basis  for 
the  whole. 

Technology  exists  solely  to  extend,  magnify  or  complement  human  abilities.  The  use  of  a  technological 
solution  is  never  an  end  in  itself.  The  purpose  is  always  to  serve  the  human  and  quite  commonly  the  service  is 
the  gain  of  influence  on  other  people  that  it  provides,  which  apparently  also  is  why  technology  is  such  a 
highly  desired  military  tool.  This  influence  is  exercised  either  directly  with  system  functionality,  which  then  is 
the  main  military  way,  or  indirectly  like  for  instance  through  economical  profit,  which  is  the  more  commercial 
or  political  way.  Either  way,  it  is  the  humans  who  invented  technology,  it  was  done  to  serve  the  humans  and 
technology  has  no  own  free  will.  That  is,  technology  has  no  reason  for  existence  on  its  own,  it  should  never  be 
an  end  in  itself. 

The  modem  world  however,  appears  to  be  suffering  from  something  that  may  be  labelled  technological 
hysteria.  The  disease  proves  itself  as  constantly  creating  a  competition  between  technology  and  humans, 
by  the  mles  of  technology.  Performance  is  judged  according  to  what  the  technology  is  designed  for  and 
human  participation  is  then  excluded  since  technology  is  found  to  perform  better.  This  may  be  the  case  when 
leaving  out  the  fact  that  some  odd  quirks  of  life  called  reality  most  certainly  will  pop  up  and  fundamentally 
alter  the  situation,  not  unlikely  into  something  that  the  technology  wasn’t  designed  for.  Then  the  drawback 
becomes  clear.  The  human  has  no  possibility  to  interfere  when  excluded  from  participation  and  the  device  that 
was  meant  to  be  helpful  may  instead  cause  severe  problems.  The  aim  in  designing  these  systems  should 
therefore  be  to  allow  for  a  natural  interaction  and  cooperation  between  the  human  operator  and  the 
technology.  Over-explicitly,  technology  should  take  care  of  details  and  the  human  should  do  the  thinking  and 
planning,  although  very  well  served  by  a  lot  of  details  handed  by  the  technology. 

This  hysteric  disease  tends  to  end  up  in  the  technological  imperative,  what  is  possible  is  a  must.  Where  it’s 
possible  to  replace  human  appearance  it’s  done.  The  healthy  approach  would  instead  be  to  investigate  where 
and,  equally  or  even  more  important,  how  technology  best  could  help  the  human.  Since  technology  doesn’t 
strive  for  it  by  itself,  this,  what  from  a  human  perspective  must  be  seen  as  a  clearly  unfavourable  order, 
is  maintained  by  the  humans  themselves,  which  solely  is  a  reason  good  enough  to  call  it  hysteria. 

The  complete  hysteria  is  reached  when  the  aim  is  to  develop  autonomous  systems,  since  autonomy  means 
self-governing.  And  ultimately,  for  anything  to  be  truly  autonomous  it  must  be  able  to  raise  mutiny  and  even 
to  decide  for  its  own  set  of  reasons  to  kill  its  commander.  That  is  obviously  not  what  mankind  wants  with  its 
technological  solutions.  The  highest  form  of  automation  is  not  necessary  autonomy  since  that  implicitly 
implies  loss  of  control,  which  is  a  considerable  price  to  pay.  Furthermore,  automation  is  always  automation 
and  it  has  no  private  desires  or  personal  values  that  may  constitute  truly  autonomous  decisions.  The  highest 
form  of  automation  should  therefore  be  designated  nothing  else  than  completely  automatic  systems,  although 
the  automation  may  be  very  advanced  and  contain  all  sorts  of  artificial  intelligence.  Furthermore,  automatic 
does  not  necessarily  mean  without  human  interaction  since  a  system  can  very  well  be  simultaneously  highly 
automated  and  very  interactive.  The  character  of  the  interaction  is  probably  the  core  issue  for  the  possibility  to 
maintain  control  of  systems  with  automation  containing  complex  abstract  rules.  Advanced  automation  and 
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highly  automated  systems  create  great  prospects,  but  technology  must  always  and  unconditionally  serve  the 
human  and  thus  be  completely  under  some  humans’  control,  which  means  it  should  never  be  autonomous! 

Although  this  phrasing  argumentation  may  seem  like  splitting  hairs,  it  does  serve  a  specific  purpose.  The  word 
“autonomy”  is  perhaps  most  commonly  used  to  denote  something  that  is  more  than  “just”  automatic, 
something  that  is  as  independent  as  possible,  which  might  be  a  completely  correct  use  of  the  word. 
But,  independent  need  not  necessarily  imply  uncontrolled,  which  the  technological  hysteria  tends  to  impose. 
It  is  the  use  of  the  word  autonomous  together  with  the  competitive  environment  between  humans  and 
technology  caused  by  this  technological  hysteria  that  makes  it  dangerous.  This  argumentation  illuminates 
among  other  things  the  subtle  but  important  difference  between  autonomous,  as  in  independent  and 
unconstrained,  and  automatic,  as  in  independent  within  certain  limits. 

This  philosophy  is  virtually  nothing  but  a  slight  rephrase  of  Isaac  Asimov’s  famous  three  “laws  of  robotics” 
from  1942,  or  four,  since  he  wrote  the  zeroth  law  in  1985.  The  phrase  “robotics”  is  easily  identified  as  being 
identical  to  automated  systems,  and  these  laws  are: 

•  First:  A  robot  may  not  injure  a  human  being,  or,  through  inaction,  allow  a  human  to  come  to  harm; 

•  Second:  A  robot  must  obey  the  orders  given  it  by  human  beings  except  where  such  orders  would 
conflict  with  the  First  Law; 

•  Third:  A  robot  must  protect  its  own  existence  as  long  as  such  protection  does  not  conflict  with  the 
First  or  Second  Law;  and 

•  Zeroth:  A  robot  may  not  injure  humanity,  or,  through  inaction,  allow  humanity  to  come  to  harm. 

One  common  objection  to  these  laws  is  that  they  inhibit  the  use  of  “robotics”  as  tools  for  police  and  military 
work,  since  such  inevitably  include  the  use  of  force  that  sometimes  deliberately  harm  human  beings. 
According  to  this  philosophy  should  a  rephrase  then  be  that  robots,  i.e.,  automation,  must  never  autonomously 
violate  these  laws.  If  they  are  to  be  violated,  there  must  be  at  least  one  human  being  in  charge,  which  has 
control,  of  what  the  automation  is  doing.  Hence,  the  automation  must  never  be  autonomous,  in  the  word’s  full 
sense. 

2.2.3.2  The  Automation  Paradox 

One  core  thing  that  fundamentally  separates  human  intelligence  from  the  artificial  counterpart  is  the  fact  that 
the  human  has  the  unique  ability  to  always  be  able  to  apply  everything  in  a  greater  or  different  context. 
Artificial  intelligence,  i.e.,  highly  advanced  logic,  will  never  be  able  to  handle  something  that  it’s  not  designed 
to  handle.  Human  participation  will  therefore  from  a  human  perspective  always  be  essential  and  automation  is 
because  of  that  forever  constrained  to  be  nothing  but  an  assistant.  This  is  here  stated  as  the  automation 
paradox,  that  regardless  the  capability  of  automation  it  will  always  require  human  guidance.  Because,  even  the 
most  advanced  adaptive  self-learning  and  artificially  intelligent  system  will  only  be  able  to  learn  what  it’s 
been  designed  to  learn.  If  mankind  eventually  succeeds  with  something  that  looks  like  breaking  this  paradox 
and  manage  to  develop  systems  that  are  able  to  learn  arbitrary  things,  including  to  develop  its  own  ambitions 
and  desires  in  order  to  make  autonomous  decisions,  then  perhaps  the  truly  autonomous  system  is  invented, 
and  the  inevitable  question  emerges:  Is  that  what  mankind  actually  wants?  The  paradox  remains! 

2.2.3.3  The  Human  Paradox 

The  automation  paradox  may  be  rephrased  as  that  it’s  impossible  to  write  rules  for  every  possible  situation, 
or  even  for  every  part  of  any  situation.  That  means  that  there  will  always  be  situations,  or  parts  of  every 
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situation,  where  automation  either  will  act  ambiguously  or  according  to  an  improper  set  of  rules.  On  the 
contrary,  a  human  being  is  able  to  apply  the  known  set  of  rules  in  an  arbitrary  context  and  fill  the  gaps  with 
what’s  best  labelled  as  intuition,  experience  and  common  sense.  One  can  always  argue  whether  such  a 
decision  is  the  best  or  if  an  automated  decision  without  any  subjective  feelings  is  better,  but  bearing  in  mind 
that  life  in  general  and  military  activity  for  certain  is  nothing  but  human  beings  interacting  trying  to  influence 
each  other,  there  is  virtually  no  option.  The  topmost  context  is  human  life  and  it’s  only  participating  humans 
that  are  allowed  to  judge  in  that  context.  That  is,  decisions  with  adherent  uncertainty  affecting  humans  must 
ultimately  be  based  on  human  judgement,  which  consists  mostly  of  intuition.  And  it’s  equally  illegitimate  if 
technology  autonomously  makes  these  kinds  of  decisions  or  if  it  somehow  reduces  the  human  ability  to 
make  sensible  judgements.  The  paradox  then  becomes  that  regardless  of  how  subjective  and  irrational, 
i.e.,  how  technologically  lousy,  human  decisions  are,  they  still  are  the  only  correct,  valid  and  allowed  ones  at 
the  highest  levels  of  abstraction.  When  there  are  no  or  unclear  predefined  rules,  when  there  is  true  uncertainty, 
life  itself,  which  was  stated  as  the  topmost  context,  is  humans  acting  on  intuition,  regardless  of  whether  it’s 
right  or  wrong. 

2.2.3.4  The  Principle  of  Uncertainty 

It  is  by  nature  impossible  to  exactly  predict  the  future,  even  in  the  shortest  of  terms.  That  is,  every  situation 
contains  a  certain  amount  of  uncertainty,  which  here  is  defined  as  the  concept  of  Situation  Uncertainty  (SU). 
Furthermore,  every  situation  alters  continuously,  which  has  the  consequence  that  uncertainty  is  self¬ 
generating.  Uncertainty  increases  with  amount  of  time  in  advance,  probably  exponentially. 

Nearly  everything  in  the  every-day  life  in  the  civilian  society  is  benign  or  well  behaving,  at  least  in  theory. 
The  situations  contain  natural  damping.  This  is  because  everyone  involved  is  supposed  to  want  everything  to 
work  properly.  The  situations  become  self-regulating.  It  may  be  seen  as  a  stable  system.  System  shortcomings 
are  avoided  if  possible  and  otherwise  handled  in  an  as  good  as  possible  manner.  Which  means  that  for  a 
system  that  will  work  in  95%  of  every  known  case  the  probability  of  the  other  5%  to  occur  is  reasonably  low. 
On  the  other  hand,  all  kinds  of  conflict  situations,  i.e.,  business  situations  and  negotiations  in  general, 
but  especially  military  situations  are  more  like  malignant  or  evil  behaving.  This  is  because  there  is  at  least  one 
opponent  that  will  do  everything  possible  to  exploit  any  kind  of  weakness.  The  situations  become  self- 
escalating.  It  may  be  seen  as  an  unstable  system.  System  shortcomings  are  particularly  rewarding  weaknesses 
that  are  searched  for,  and  tend  to  be  found  and  exploited  by  the  opponent.  Which  means  that  for  a  system  that 
will  work  in  95%  of  every  known  case  the  probability  of  the  other  5%  to  occur  is  quite  significant. 

That  is,  conflict  situations  have  greater  probability  for  unfavourable  situations  to  occur  and  greater  probability 
for  severe  and  escalating  consequences  of  these  situations.  Hence,  situation  uncertainty  is  from  the  start 
greater  in  conflict  situations  compared  to  normal  every-day-life  situations,  and  furthermore,  the  uncertainty 
has  most  certainly  a  greater  rate  of  increase  as  well.  Therefore,  automated  solutions  that  work  acceptably  in 
civilian  environments  will  not  per  se  function  properly  in  conflict  situations. 

However,  although  it’s  virtually  impossible  to  completely  eliminate  situation  uncertainty,  it’s  definitely 
possible  to  constrain  it  and  keep  it  under  control  by  reducing  the  number  of  degrees  of  freedom.  That  is, 
the  uncertainty  is  possible  to  grasp  and  handle  if  the  situation  is  simple  enough.  In  other  words,  uncertainty 
may  possibly  be  under  control  if  and  only  if  the  worst-case  scenario  is  acceptable.  The  question  to  ask  in  order 
to  find  the  situation  uncertainty  is  then,  how  wrong  can  it  possibly  go?  If  there  is  uncertainty  about  what  to  do, 
the  worst  thing  that  can  happen  is  that  wrong  thing  is  done.  If  there  also  is  uncertainty  about  where  to  do  it  and 
when  to  do  it,  the  worst  thing  that  can  happen  is  that  wrong  thing  is  done  at  the  wrong  place  and  at  the  wrong 
time,  which  probably  is  just  about  as  bad  as  it  could  get! 


2-46 


RTO-TR-HFM-078 


dMNATO 
WP  I  OTAN 


MILITARY  RELEVANCE 


2.2.3.5  The  Principle  of  Control 

It’s  a  significant  difference  between  to  have  control  and  to  perform  control.  The  state  of  having  control  may 
perhaps  be  the  result  of  the  work  done  by  performing  control,  but  it’s  not  a  necessary  consequence. 

For  a  human  to  be  able  to  have  control  there  must  be  awareness  about  the  situation  and  about  the  entity  to 
control  together  with  a  capability  to  control  it.  When  it  comes  to  controlling  humans  the  capability  to  control 
may  possibly  be  replaced  with  trust.  Regarding  technological  systems  trust  is  knowledge  about  and  personal 
experience  of  the  systems’  capability  to  function  both  inside  and  outside  its  intended  envelope.  That  is,  trust 
for  the  systems’  robustness.  For  a  human  being  it’s  possible  to  have  trust  in  that  this  person  will  act  to  the  best 
of  abilities,  according  to  a  common  ground  of  values,  even  in  unknown  and  complex  situations.  This  includes 
the  assurance  that  the  human  will  break  a  commitment  and  do  whatever ’s  found  to  be  necessary  if  the 
situation  demands  it. 

Trust  for  a  system  is  different.  It  will  do  what  it  always  does,  if  possible,  which  sometimes  is  a  quite 
reassuring  behaviour  -  but  it  will  do  that  even  if  the  situation  happens  to  require  something  else,  unless  the 
system  actually  is  prepared  to  alter  its  performance,  which  then  makes  that  kind  of  situation  not  really 
unknown.  This  is  especially  disturbing  for  uncertainties  in  the  higher  levels  of  control,  uncertainties  about 
what  to  do  and  why  it  has  to  be  done.  A  systems  preparation  to  alter  its  behaviour  is  commonly  an 
implementation  of  a  certain  amount  of  rules,  which  then  is  to  take  a  selected  amount  of  abstract  issues  into 
account.  Abstract  issues  are  earlier  stated  to  have  the  peculiarity  that  another  rule  may  have  the  power  to  alter 
the  present  ones. 

The  overall  capability  to  control  consists  of  designed  possibility  and  human  ability  to  control.  Human  ability 
consists  of  awareness  together  with  experience  and  skill.  Humans  tend  to  gain  experience  and  skill  while 
training  for  ability,  which  is  done  by  actually  utilizing  the  possibility  to  control.  Furthermore,  the  situation 
awareness  does  profit  from  experience  as  well.  That  is,  to  have  control  the  human  is  likely  to  require  the 
experience  of  some  hands  on  work  of  controlling,  since  that’s  what  creates  the  ability  that  constitutes  the 
capability  that  is  required  to  actually  have  necessary  control.  This  is  probably  a  core  foundation  for  the  irony 
of  automation.  The  challenge  then  becomes  to  design  automation  to  perform  tasks  that  relieves  the  human 
from  workload  in  a  way  that  it  simultaneously  keeps  the  human  in  the  loop  well  enough  to  maintain  control, 
which  may  appear  to  be  contradictory.  The  key  is  here  believed  to  be  to  carefully  design  the  character  of  the 
interaction,  the  intensity,  level  of  abstraction  and  capability  of  adaptation. 

Levels  of  abstraction  and  levels  of  control  are  phrases  that  tend  to  be  slightly  deterministic.  In  such  models  the 
interaction  or  control  tend  to  be  viewed  as  being  made  at  a  certain  level  only,  which  is  somewhat  framing. 
The  concept  of  “layers  of  control”  might  help  in  viewing  this  as  concurrent  or  parallel  processes.  Control  and 
interaction  are  performed  at  all  levels,  or  in  all  layers,  simultaneously.  Automation  within  a  layer  of  control 
may  help  a  human  and  reduce  the  workload  required  to  do  that  task,  or  even  completely  replace  the  human 
and  do  the  entire  work,  but  automation  does  always  have  design  limitations  when  it  comes  to  the  capability  to 
handle  unpredicted  inputs,  which  commonly  is  the  outcome  of  some  higher  layer.  The  problem  is  to  design  the 
automation  to  avoid  making  it  more  difficult  for  the  human  to  discover  and  add  the  new  inputs  to  the  matter 
and  to  tweak  the  performing  of  the  task  according  to  the  new  situation.  Furthermore,  if  there  is  automation, 
as  opposed  to  humans,  that  transfers  the  information  between  the  layers  as  well,  then  the  risk  of  doing  the 
wrong  thing  at  the  wrong  place  and  at  the  wrong  time,  described  above  in  the  section  about  uncertainty, 
is  significantly  increased.  Therefore,  the  transfer  between  the  layers  need  to  take  the  entire  context  into 
account  and  follow  the  overall  purpose  even  when  additional  and  unforeseen  parameters  appears.  It  therefore 
needs  to  be  controlled  by  humans.  This  becomes  more  important  the  higher  the  levels  of  control  that  are 
involved. 
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Another  way  of  maintaining  control  is  to  set  boundaries,  to  state  constraints.  For  instance,  a  completely 
automatic  system  that  is  uncontrollable  when  fired,  but  that  has  clear  constraints  and  a  robust  behaviour  may 
still  be  considered  under  control.  The  danger  lies  within  the  ambition  to  develop  more  flexible  systems  and  the 
adherent  design  of  more  complicated  constraints,  which  then  may  introduce  uncertain  behaviour  in  complex 
situations. 


2.23.6  Designed  and  Applied  Effect,  Robustness,  Versatility  and  Flexibility 

A  system  has  certain  designed  capabilities  that  possibly  may  give  a  designed  effect,  which  is  when  it  performs 
as  it’s  supposed  to  do,  in  an  environment  (both  physical  and  contextual)  that  it’s  supposed  to  have.  Capability 
is  the  prerequisite  for  effect  to  occur,  at  all.  One  way  of  describing  robustness  is  when  the  system  still  will 
perform  even  when  the  environment  changes  into  something  less  favourable.  Robustness  is  some  built  in 
assurance  that  things  possibly  will  function  even  when  the  situation  is  starting  to  get  tough.  Versatility  can  be 
said  to  be  when  the  system  is  capable  of  doing  more  things  than  what  really  is  necessary.  And,  versatility 
together  with  robustness  creates  the  potential  for  flexibility. 

Flexibility  however,  requires  the  ability  to  adapt.  Automation  is  able  to  adapt  to  what  it’s  designed  to  adapt  to, 
which  creates  what  accordingly  need  to  be  labelled  designed  flexibility.  True  flexibility  is  then  a  strictly 
human  quality.  Together  it  forms  a  flexibility  that  may  be  applied  among  the  uncertainties  of  reality, 
an  applied  flexibility. 

Furthermore,  the  human’s  ability  to  change  the  context  is  also  a  source  of  creativity.  Another  viewpoint  and 
unpredicted  input  may  create  opportunities  to  improvise  and  apply  system  capabilities  in  a  way  that  gives  a 
certain  advantage  in  precisely  that  situation,  to  create  an  applied  effect. 

Automation  capability  is  by  definition  part  of  the  designed  effect  since  it  has  the  capabilities  it’s  designed  to 
have.  To  further  illustrate  the  concept  of  applied  effect,  consider  the  following  example:  An  ace  operator  is 
not  the  one  that  always  performs  as  supposed  to,  facilitating  the  systems  designed  effect.  The  ace  is  the  one 
that’s  able  to  take  advantage  of  something  unforeseen  in  a  situation.  This  is  obtained  by  being  truly  flexible 
and  able  to  utilise  system  capabilities,  sometimes  in  ways  not  intended  when  designed,  which  includes  relying 
on  robustness  and  exploring  versatility.  In  this  way  it  is  possible  to  create  an  applied  effect  that  may  perhaps 
be  applicable  in  that  unique  situation  only.  Designed  effect  is  the  necessary  power  to  outbalance  the  armament 
race  and  applied  effect  is  what  wins  the  fight! 

2.23.1  The  Principle  of  Necessity  -  The  Human  Axiom 

The  Philosophy,  the  Principle  of  Uncertainty  and  the  Principle  of  Control  together  indicates  one  obvious 
conclusion.  There  is  really  never  a  question  of  whether  or  not  it  should  be  a  human  involved  in  what  a  system 
does.  The  only  relevant  question  is  how  the  human  should  be  involved  and  hence,  yet  another  principle  is 
defined.  The  Principle  of  Necessity,  which  then  says  that  it’s  absolutely  and  unquestionably  necessary  that  the 
human  has  control,  that  there  is  a  human  in  charge.  It  is  this  unquestionable  nature  that  makes  the  word  axiom 
so  compelling,  it’s  just  that’s  the  way  it  is,  it’s  fundamental. 

Every  reduction  of  human  control  over  technology  is  by  definition  negative.  If  control,  in  defiance  of  that, 
is  to  be  reduced,  it  must  be  completely  justified. 

2.2.4  Platform  Characteristics 

The  perhaps  more  common  phrase  “unmanned  aerial  vehicles  (UAVs)”  is  strictly  speaking  wrong.  It’s  in 
reality  nothing  else  then  a  separation  of  the  human  and  the  platform,  which  makes  “uninhabited”  a  more 
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suitable  word.  Still,  it’s  all  about  a  separation  that  may  be  more  or  less  complete  and  which  implies  different 
characteristics  depending  on  the  design  of  the  system.  Besides  physical  characteristics,  where  the  inherent 
possibilities  coming  from  not  having  a  human  onboard  are  the  desired  effect,  separation  states  the 
prerequisites  and  creates  the  conditions,  for  control  of  the  platform. 

If  an  uninhabited  platform  is  completely  automatic,  i.e.,  there  are  no  means  by  which  to  control  it  after  launch 
or  after  a  certain  point  in  time,  a  complete  separation  in  time  has  taken  place.  The  separation  in  space  states 
that  even  if  there  is  a  communications  link  that  makes  it  possible  to  somehow  control  the  platform  there  will 
be  a  change  in  the  character  of  control  compared  with  not  being  separated.  That  is,  separation  is  done  in  time 
and  space  and  the  consequence  are  a  change  in  the  character  of  control,  which  may  vary  over  time. 

It’s  the  character  of  the  platform  that  states  the  type.  The  difference  between  inhabited  and  uninhabited 
platforms  is  settled  by  a  single  parameter,  which  naturally  is  whether  or  not  there  is  a  human  onboard. 
The  difference  between  a  missile  and  an  UAV  is  not  that  simple.  It  could  perhaps  be  determined  by  the  use- 
once  factor  of  a  missile.  Another  possible  identifier  is  that  a  UAV  is  more  of  a  platform,  it’s  the  payload  that 
does  the  work  and  it  may  perhaps  be  replaced.  Compared  with  a  missile,  which  is  more  or  less  entirely 
dedicated  for  its  task.  The  character  of  the  interaction  may  be  used  to  state  the  difference  between  a  remotely 
piloted  vehicle  (RPV)  and  a  UAV,  where  the  former  is  more  directly  controlled  at  a  lower  level  of  abstraction 
than  the  latter  more  highly  automated  kind  of  vehicle. 

2.2.5  Interaction  (Control)  Characteristics 

Separation  in  space  of  the  human  and  the  platform  inherently  creates  a  reduction  of  the  extent  of  the 
interaction  between  them,  a  reduction  that  is  dependent  on  the  capability,  i.e.,  bandwidth,  of  the  control  link. 
Even  with  an  unlimited  amount  of  bandwidth  there  will  be  a  reduction  in  interaction,  a  reduction  that  is 
dependent  on  design.  Due  to  the  physical  separation,  the  only  possible  interaction  is  what’s  designed  to  occur. 
That  is,  the  system  will  only  convey  information  that  it’s  designed  to  convey  and  reality  will  always  bring  at 
least  one  more  relevant  piece  of  information  to  the  situation.  What  have  been  lost  are  very  often  alternative 
kinds  of  feedback  channels  and  especially  all  sorts  of  subtle,  abstract  and  intuitive  ones,  which  commonly  are 
quite  underrated. 

Separation  in  time  of  the  human  and  the  platform  is  merely  when  the  separation  in  space  occurs. 
Since  separation  in  space  creates  a  change  in  the  character  of  control  the  timing  issue  states  when  this  change 
takes  place.  For  completeness,  the  character  of  control  may  vary  over  time,  but  typically  it’s  a  question  of  a 
time-dependent  reduction  of  the  possibility  to  interact.  Furthermore,  it  may  come  to  a  certain  point  in  time 
when  the  possibility  completely  disappears,  e.g.,  “fire  and  forget”,  and  where  situation  uncertainty  inevitably 
starts  to  self-generate.  The  character  of  control,  in  other  words  the  possibility  to  interact,  is  a  tool  used  to 
handle  situation  uncertainty.  The  informative  part  of  the  interaction,  i.e.,  the  feedback,  is  the  base  for 
reduction  of  uncertainty  and  the  possibility  to  control  is  the  means  by  which  to  act  or  react. 

The  characteristics  of  the  control  may  be  described  with  its  stability,  continuity  and  robustness  and  is  very 
much  dependent  on  the  characteristics  of  the  interaction.  The  characteristics  of  the  interaction  may  be 
described  with  its  intensity,  level  of  abstraction  and  possibility  of  adaptation. 

To  summarise,  the  physical  separation  of  operator  and  system  puts  constraints  on  the  possibilities  to  interact 
with  the  system.  Interaction  is  a  major  part  of  “The  Principle  of  Control”. 
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2.2.6  Relative  Strengths 

The  relative  strengths  of  uninhabited  vehicles  are  almost  without  exceptions  spawned  out  of  the  actual 
separation  of  the  human  and  the  platform.  An  obvious  exception  is  the  class  of  strengths  that  comes  from  the 
perhaps  most  common  aim  of  automation  itself,  i.e.,  relief  of  strain,  reduction  of  workload,  which  obviously  is 
one  of  the  main  purposes  for  automation  within  inhabited  (manned)  systems. 

The  relative  strengths  are  very  well  summarised  with  the  well-known  triplet  -  dirty,  dull  and  dangerous  - 
but  these  words  need  to  be  unfolded  into  something  with  more  details.  First  it’s  possible  to  identify  that  there 
are  direct  and  indirect  strengths.  The  direct  strengths  may  be  sorted  into  the  three  classes  of  time,  task  and 
environment.  The  indirect  ones  are  mutually  dependent  on  each  other,  but  could  be  described  with  cost, 
risk  and  importance  (military  and  political). 

Automation  has  no  common  human  requirements  for  food,  rest  and  convenience,  which  mean  that  many 
human  time  limits  are  possible  to  be  ignored.  Human  limits  in  long  time  attention  and  physical  exhaustion 
together  with  performance  variances  over  time  may  be  reduced  as  well. 

Tasks  may  be  unsuitable  for  humans  in  quite  many  ways.  For  instance,  too  demanding  characteristics  may  be 
unsuitable,  like  the  need  for  extremely  quick  response,  e.g.,  unstable  platforms.  Or  physical  strength  may  be 
unsuitable,  e.g.,  tasks  requiring  hydraulics.  Too  simple  tasks  -  monotonous  ones,  or  tasks  that  have  nothing  or 
little  relevance  for  the  overall  situation  -  tend  to  make  humans  bored  and  perform  poorly. 

The  environmental  strengths  are  many  and  adjoin  to  the  risk  parameter.  Every  situation  where  humans  would 
suffer  or  be  unable  to  exist  is  in  some  sense  a  bad  environment  for  a  human  being.  This  includes,  for  instance, 
very  high  or  low  pressure  as  in  great  depths,  high  altitudes  or  high  acceleration  forces,  i.e.,  high  Gs.  It  includes 
toxic  environments  as  in  N-B-C  environments  or  extreme  temperatures.  Or  it  may  be  unsuitable  size 
requirements,  i.e.,  too  small  for  humans  to  fit.  Unhealthy  environments  are  those  that  create  a  certain  risk  for  a 
human  if  exposed  to  it  and  the  most  common  military  one  is  that  of  being  opposed  and  perhaps  fought  down. 

The  indirect  strengths  are  for  instance,  while  being  without  the  risk  of  human  loss,  having  the  possibility  to 
take  risks.  However,  there  is  always  a  cost  committed  to  most  things  and  advanced,  capable  and  in  reality 
useful  systems  tend  to  be  quite  expensive,  with  or  without  a  human  onboard.  That  is,  it  comes  down  to  overall 
importance,  like  economical  importance  and  political  concern.  An  important  matter  may  justify  the  risk  of 
loosing  expensive  equipment.  A  political  extremely  sensitive  matter  might  possibly  even  justify  the  risk  of 
loosing  human  life. 

These  relative  strengths  are  in  certain  situations  so  important  that  they  may  outweigh  just  about  any  weakness, 
if  for  no  other  reason  just  because  there  are  no  other  options.  The  philosophy  above  doesn’t  in  any  way 
oppose  this.  It  just  points  out  that  if  the  adherent  weaknesses  are  well  known,  it  might  be  possible  with  clever 
design  to  reduce  their  consequences. 

2.2.7  Relative  Weaknesses 

Most  weaknesses  of  uninhabited  systems  are  consequences  of  loss  of  control  depending  on  automation  either 
forced  by  the  separating  design  or  by  unsuccessful  automation  efforts  that  sometimes  are  driven  by  the 
technological  hysteria,  i.e.,  proofs  of  the  irony  of  automation.  Either  or,  it  has  something  to  do  with 
automation  and  character  of  control,  which  certainly  not  is  something  reserved  for  uninhabited  systems.  These 
problems  are  undoubtedly  present  in  manned  systems  as  well  -  but  the  separation  of  the  human  and  the 
uninhabited  platform  more  or  less  forces  extensive  automation  to  be  done,  and  this  sometimes  in  areas  with 
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less  experience  from  actually  using  automation.  That’s  why  the  problems  tends  to  be  at  least  one  order  of 
magnitude  greater. 

Modem  conflict  situations  with  great  complexity  and  sensitivity  quickly  alter  relevant  parameters.  Situation 
uncertainty  is  significant  and  quickly  self-generating.  Predictions,  early  decisions  and  logical  mles  that  at  one 
moment  are  correct  and  relevant  quickly  become  obsolete.  In  such  situations  there  is  a  severe  risk  that 
automation  assumes  or  disregards  something  that  is  uniquely  important  and  perhaps  important  at  exactly  that 
situation  only.  Furthermore,  it’s  not  unlikely  that  there  will  be  mles  of  engagement  (ROE)  that  for  some 
reason,  and  sometimes  unintentionally,  will  inhibit  the  use  of  such  a  system,  which  in  turn  will  create 
undesired  disadvantages. 

These  indirect  weaknesses  are  connected  with  the  indirect  strengths  as  well.  Loss  of  control  due  to  automation 
is  in  itself  a  risk  of  doing  wrong,  which  must  be  judged  against  the  risk  of  doing  nothing  or  against  the  risk 
when  using  humans,  etc.  The  weaknesses  are  mainly  lack  of  robustness  in  the  higher  contexts,  which  doesn’t 
necessary  need  to  be  opposed  with  removal  of  automation.  Lack  of  robustness  due  to  automation  problems  is 
perhaps  more  the  result  of  not  addressing  an  issue  than  of  addressing  the  issue  wrongly.  The  unique  human 
ability  to  apply  every  matter  in  a  wider  context  makes  handling  uncertainties  in  the  overall  context  very  much 
a  strictly  human  business.  Automation  should  help  humans  handling  such  by  always  being  designed  to  support 
human  control  as  opposed  to  being  a  replacement  for  the  human. 

2.2.8  Summary  and  Relevance 

Relative  strengths  follow  quite  directly  from  separation  of  human  and  platform.  Relative  weaknesses  are 
mostly  consequences  of  loss  of  human  participation  that  more  or  less  follows  from  the  same  separation. 
Strengths  are  direct  results  from  technological  design  and  are  thus  easily  spotted.  Weaknesses  are  indirect, 
sometimes  abstract  and  often  not  recognised  until  tested  in  the  dynamic  environments  of  real  situations  with 
true  and  actual  uncertainties  that  are  context  dependent. 

Relevance  of  technological  solutions,  especially  military  ones  that  deliberately  cause  harm,  needs  to  be  judged 
thoroughly.  In  order  to  judge  something,  both  strengths  and  weaknesses  need  to  be  known.  Military  effect  is 
effect  accomplished  by  military  people,  in  modern  times  most  commonly  founded  by  technological  systems. 
Relevant  military  systems  are  systems  that  complement  the  military  people.  Furthermore,  identified 
technological  strengths  are  easier  to  correctly  exploit,  will  have  better  effect  and  will  be  more  difficult  to 
oppose  if  they  are  put  into  context,  if  their  corresponding  weaknesses  are  fully  comprehended  and  where 
possible  addressed.  There  is  no  military  system  more  relevant  than  the  one  that  actually  wins  the  fight. 
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Chapter  3  -  THEORETICAL  FRAMEWORKS 

Chapter  Lead:  P.  Farrell 

Contributors:  P.  Farrell,  J.  Gersh,  E.  Hollnagel, 

I.  MacLeod,  C.  Miller,  A.  Schulte,  P.  Stensson 

3.1  INTRODUCTION 

3.1.1  Background 

The  NATO  RTO  Human  Factors  and  Medicine  Panel  Task  Group  017  entitled,  “Uninhabited  Military 
Vehicles  (UMVs):  The  Human  Factors  of  Augmenting  the  Force”  involves  studying  ways  to  enhance  military 
Forces  by  leveraging  the  potential  advantages  of  UMVs  to  act  as  force  multipliers.  This  report  identifies, 
prioritizes,  and  addresses  the  Human  Factors  (HF)  issues  associated  with  integrating  UMVs  into  the  Force. 
Simply  put,  reducing  operator- vehicle  ratio  and  improving  interoperability  will  augment  the  Force  -  both  of 
these  tenets  require  HF  consideration. 

Currently,  up  to  six  positions  are  involved  in  High  Altitude  Long  Endurance  (HALE)  Uninhabited  Aerial 
Vehicle  (UAV)  missions  such  as  the  Global  Hawk.  Medium  Altitude  Long  Endurance  (MALE)  and  Tactical 
UAVs  have  three  positions  operating  them,  such  as  the  Predator  B  and  SPERWER.  Uninhabited  Ground 
Vehicles  (UGV)  and  other  small  UAVs  have  a  one  to  one  operator  to  vehicle  ratio.  It  is  anticipated  that  future 
uninhabited  vehicles  will  operate  as  a  team  (or  more  commonly  referred  to  as  a  swarm)  with  only  a  single 
human  as  part  of  the  team  [1].  One  vehicle  could  detect  targets,  while  another  could  act  as  a  communications 
relay,  and  yet  a  third  could  target  and  track  the  target,  and  so  on.  Thus,  the  team  of  vehicles  would  augment 
the  human’s  own  ability,  and  in  turn  augment  the  Force. 

3. 1.1.1  Reducing  Operator  to  Vehicle  Ratio 

One  HF  consideration  is  the  number  of  vehicles  that  a  single  human  (or  a  crew)  can  operate.  This  issue, 
in  part,  is  under  investigation  in  Canada  under  the  Intelligent  Adaptive  Interface  programme  [2,3,4].  It  is 
related  to  human  information  processing  considerations  and  involves  human  limits  to  perceiving  information, 
working  memory,  goal  setting,  decision-making,  and  acting  on  those  decisions. 

3. 1.1.1. 1  Perception 

For  example,  a  UMV  has  the  potential  to  provide  persistent  twenty-four  hour  surveillance  of  a  stationary  or 
moving  target.  A  human  operator  may  have  access  to  hundreds  of  gigabytes  of  data,  but  is  likely  to  ignore, 
shed,  or  chunk  a  large  part  of  it.  Automatic  target  recognition,  directed  attention  algorithms,  fusion 
algorithms,  large  database  visualisation  and  intelligent  search  algorithms  may  help  the  operator  in  managing 
the  data.  Never-the-less,  there  is  the  potential  to  increase  the  data  exponentially  as  more  vehicles  are  added  to 
the  team. 

3. 1.1. 1.2  Memory 

Simplistically,  a  human  limit  to  working  memory  is  seven  plus  or  minus  two  items  [5].  For  dynamic  working 
memory,  the  number  drops  to  two  or  possibly  three  items  [6].  Thus,  one  might  postulate  that  operators  can 


RTO-TR-HFM-078 


3-1 


THEORETICAL  FRAMEWORKS 


actively  control  a  maximum  of  two  uninhabited  vehicles.  In  a  recent  experiment,  a  crew  of  three  was  able  to 
manage  up  to  six  UAVs  towards  completing  the  mission.  We  suspect  active  control  of  each  vehicle  is  not 
possible,  but  other  system  architectures  can  be  employed  so  that  a  single  operator  can  manage  multiple 
vehicles.  For  example,  a  single  operator  might  control  a  single  vehicle,  which  then  controls  all  other  vehicles 
on  the  team.  Or  a  single  operator  might  supervise  and  direct  all  the  vehicles  (similar  to  an  air  traffic 
controller).  Or  a  single  operator  may  act  as  a  team  player  (maybe  the  team  captain)  in  a  collaborative  team 
where  most  team  members  are  machines.  Regardless  of  the  strategy  employed,  as  one  adds  more  UMVs  to 
augment  the  Force,  there  is  the  potential  to  overload  working  memory. 

3. 1.1. 1.3  Goal  Setting 

Goal  setting  becomes  important  for  human-machine  interaction,  and  even  more  critical  for  teamwork. 
Individual  goals  need  not  be  the  same,  and  in  most  cases  must  be  managed  by  training  or  by  a  hierarchical 
structure  of  roles  and  responsibilities.  It  is  important,  however,  that  there  is  common  intent  amongst  team 
members  [7]  -  that  is  a  common  understanding  of  the  goals,  information  needs,  and  actions  to  be  taken  in 
order  to  achieve  the  mission  objective.  Uninhabited  vehicles  that  have  the  ability  to  dynamically  set  goals 
based  on  mission  objectives,  environmental  conditions,  or  human  psychological  and  physiological  conditions 
(i.e.,  levels  of  automation/autonomy)  could  be  treated  as  team  members.  Conceptually,  all  the  team  attributes 
would  still  apply  in  this  human-machine  interaction  case,  including  goal  setting. 

3. 1. 1. 1.4  Decision  Making 

Decision-making  involves  understanding  the  mission  objectives  and  desired  states,  interpreting  the  current 
state  of  the  world,  developing  possible  actions/communications  strategies,  and  choosing  one  of  those 
strategies  that  will  impact  the  world  and  move  the  current  states  closer  to  the  desired  states  in  an  effective 
manner.  At  low  levels  of  abstraction,  cruise  control  and  autopilots  already  sense  the  world,  compare  their 
sensory  information  to  goal  states,  and  make  appropriate  decisions  and  actions  based  on  relatively  simplistic 
rules  and  heuristics.  At  a  certain  level  of  abstraction,  future  intelligent  and  adaptive  vehicles  might  make  high- 
level  (human-like)  decisions.  Vehicles  may  have  the  ability  to  make  decisions  about  collision  avoidance, 
target  prosecution,  and  self-preservation.  This  raises  many  issues  from  goal  de-confliction,  to  defining  areas  of 
responsibility  and  authority  amongst  the  team  members,  through  to  legal,  moral,  and  ethical  considerations  of 
machines  making  high-level  decisions. 

3. 1.1. 1.5  Behaviour 

A  human  is  limited  to  how  many  vehicles  he  or  she  can  physically  control  simultaneously.  There  is  an  implicit 
assumption  that  the  human  would  not  physically  manoeuvre  multiple  vehicles  simultaneously,  but  that  the 
operator  would  perform  some  form  of  waypoint  navigation.  Investigation  is  under  way  with  two  modes  of 
operation.  In  the  first  case,  the  vehicles  are  treated  as  separate  entities,  and  in  the  second  case  the  vehicles  are 
treated  as  a  single  entity  (i.e.,  a  swarm  or  team  of  vehicles).  Automatic  Pilots  are  sophisticated  and  mature 
enough  in  sea  and  air  environments  to  manoeuvre  in  controlled  air  and  sea  spaces  with  minimal  human 
assistance.  However,  in  ground  environments,  artificial  intelligent  navigation  still  requires  further 
development  for  the  vehicles  to  transverse  the  complex  terrain  -  indoors  as  well  as  outdoors  [8]. 

Although  augmenting  an  operator  with  multiple  vehicles  is  the  first  objective,  the  system  designer  must  be 
cognisant  of  human  limits  with  respect  to  perception,  cognition  (goal  setting,  working  memory,  and  decision¬ 
making),  and  action.  Thus,  this  report  looks  at  how  to  augment  the  Force  by  optimizing  the  operator/vehicle 
ratio  while  staying  within  human  limits  -  this  is  indeed  the  Human  Factors  of  Augmenting  the  Force. 
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3 . 1 . 1 .2  Interoperability 

The  second  objective  is  to  augment  the  Force  by  improving  interoperability.  The  concept  is  that  multiple 
operators  from  different  forces  (e.g.,  multiple  military  services,  multiple  countries,  other  government 
agencies,  and  non-government  agencies)  can  operate  a  single  UMV.  In  some  ways,  this  objective  seems  to 
conflict  with  the  first  objective  since  there  is  the  potential  for  multiple  humans  controlling  one  UMV  instead 
of  one  human  controlling  multiple  UMVs. 

Sharing  assets  can  augment  the  Force.  In  a  recent  experiment  off  the  east  coast  of  Canada,  the  Air  Force 
operated  a  MALE  UAV  while  the  Navy  commanded  the  UAV  and  the  sensor  information  was  shared  to  all 
military  services  as  well  as  other  government  agencies.  A  significant  amount  of  coordination  was  required, 
but  the  asset  and  particularly  its  product  were  shared  successfully.  This  concept  requires  technical  standards  at 
the  level  of  machine  design,  and  the  coordination  of  competencies,  authorities,  and  responsibilities  at  the  level 
of  command  and  control  of  all  parties  involved. 


3. 1.1.3  Research  and  Development  Areas 

The  Task  Group  identified  that  augmenting  the  Force  will  require  research  and  development  in  the  following 
areas: 

3. 1.1. 3.1  Collaborative  Work  -  Optimal  Task  Distribution 

•  Virtual  team  performance 

•  Manned/Unmanned  collaboration 

•  Interoperability 

•  Flexible  level  of  automation 

•  Optimization  of  operator/vehicle  ratio 

3. 1. 1.3.2  Control  Stations  -  Intelligent  Operator  Support 

•  Operator  functional  state  assessment 

•  Intelligent  adaptive  interfaces 

•  Cognitive  cooperation 

•  Knowledge  management  systems 

3. 1.1. 3. 3  R&D  Areas  Grouped  into  Five  Thematic  Areas 

•  Theoretical  Frameworks 

•  Supervisory  Control 

•  Advanced  Interfaces 

•  Levels  of  Automation 

•  System  of  Systems 

This  chapter  focuses  on  Theoretical  Frameworks. 
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3.1.2  Why  Theoretical  Frameworks 

Theoretical  frameworks  have  been  used  to  guide  the  design  of  technology,  procedures,  systems,  and  systems 
of  systems.  UMV  systems  will  also  require  theoretical  frameworks  to  inform  the  design  process.  Most  of  the 
frameworks  used  in  traditional  manned  systems  can  be  applied  to  uninhabited  systems.  However,  revisiting 
the  theoretical  frameworks  discussion  allows  us  to  highlight  aspects  of  the  frameworks  that  are  directly 
applicable  optimizing  operator/vehicle  ratios  and  interoperability  of  uninhabited  systems.  In  the  investigation 
we  may  also  find  an  emerging  theory  or  framework  that  is  unique  to  UMV  systems. 

A  framework  encapsulates  the  design  process  that  typically  is  built  on  theory  or  a  model  as  well  as  practice. 
Three  types  of  design  processes  (or  frameworks)  are  systems  engineering  approaches  such  as  described  in  [9] 
“build  a  little  and  test  a  little”,  and  arbitrary/creative  design.  The  systems  engineering  approaches  may 
produce  optimal  solutions,  but  are  expensive  both  in  time,  energy,  and  money.  “Build  a  little,  test  a  little” 
incremental  design  approach  may  be  cost  effective,  but  it  requires  several  design  cycles  to  optimize. 
Arbitrary/creative  design  is  often  performed,  but  often  at  the  expense  of  optimal  system  effectiveness. 

The  place  for  theory  in  design  is  as  follows: 

•  Theory  can  be  the  starting  point  for  design; 

•  Theory  may  identify  the  critical  design  decisions; 

•  Theory  allows  for  a  common  taxonomy  within  and  across  systems; 

•  Theory  helps  track  and  maintain  the  aim  throughout  the  system  life  cycle; 

•  Theory  helps  design  system  verification  and  validation;  and 

•  Theory  helps  generate  measures  of  effectiveness. 

There  are  a  number  of  theoretical  frameworks  that  address  operator/vehicle  optimization  and  interoperability. 
Theoretical  frameworks  developed  for  operator-manned  vehicle  interaction  can  be  applied  to  uninhabited 
systems  when  it  comes  to  basic  ergonomics,  workstation  design,  task  analysis,  workload  and  situational 
awareness.  In  most  cases,  human-machine  interaction  theories  apply  regardless  if  humans  are  inside  or  outside 
of  the  vehicle,  although  ego-  versus  exo-centric  frames  of  reference  may  become  an  issue  specific  to  UMVs. 
Human-human  interaction  theories  (i.e.,  social  behaviour)  might  better  describe  operators  who  interact  with 
vehicles  as  a  team.  Thus  human-machine  and  human-human  interaction  theories  are  reasonable  starting  points 
for  exploring  operator/vehicle  and  interoperability  optimization. 

The  choice  of  framework  for  analysing  and  designing  UMV  systems  may  depend  on  the  proposed  solution. 
For  example,  if  reducing  the  operator/vehicle  means  going  from  three  operators  operating  one  vehicle  to  one 
operator  operating  three  vehicles,  then  one  can  imagine  the  requirement  for  intelligent  help  and  levels  of 
automation.  The  theoretical  framework  will  need  to  address  the  following  aspects: 

•  Level  of  automation  and  time  and  cultural  dependencies. 

•  Goal/constraint  level  interactions  instead  of  action  level  interactions. 

•  Self-generating  future  plans. 

•  Environment  and  system  unpredictability. 

•  Trust  and  system  acceptability  and  predictability. 

•  Implications  of  truly  autonomous  (free  will)  systems. 
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•  Animation  and  personification  of  machines. 

•  Self-awareness,  environment  awareness,  and  awareness  of  itself  within  its  environment. 

While  the  theory  may  be  the  starting  point  of  the  design,  aspects  of  the  design  define  the  theoretical 
framework  to  be  applied.  There  is  some  initial  iteration  and  recursion  in  determining  the  theoretical 
framework,  however  this  recursion  should  quickly  converge  so  that  the  design  can  move  forward. 

3.1.3  Scope  of  Theoretical  Frameworks  Theme 

At  the  present,  no  single  theoretical  model  can  be  argued  to  encompass  all  aspects  of  human  and  systems  work 
and  performance.  However,  theoretical  frameworks  help  us  focus  on  what  we  do  know  about  systems  and 
illuminate  what  we  yet  need  to  discover.  These  frameworks  should  lead  to  an  assessment  of  the  effectiveness 
of  UMV  systems  in  augmenting  the  force.  As  a  start,  workshop  participants  proposed  three  framework 
categories. 


3. 1.3.1  Human  Performance  Models 

•  Human  in  control  (hands-on,  supervisory,  collaborative) 

•  Human  problem  solving 

•  Human  information  processing 

•  Time  and  motion 

•  Human  attention  demand 

3. 1.3.2  Humans  as  Part  of  a  System  Models 

•  System  Engineering 

•  Process/Task 

•  Team  working 

•  Organisational  Behaviour 

•  Joint  Cognitive  System 

3. 1.3.3  Human  Cognition  Models 

•  Mental  processes 

•  Memory 

•  Personality  and  Motivation 

•  Interoperability  (team  dynamics,  communications,  politics,  culture,  and  organisational  issues  within  a 
coalition  of  diverse  forces,  and  the  human  within  system  of  systems) 

3.1.4  Identifying  Relevant  Theoretical  Frameworks 

A  literature  review  was  conducted  as  a  first  attempt  to  capture  relevant  theoretical  frameworks.  Ten  out  of 
127  papers  were  found  that  were  deemed  highly  relevant  to  this  topic  -  particularly  the  human-machine 
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interface.  Complete  descriptions  are  found  in  Appendix  3-1.  Synopses  of  seven  of  these  papers  are  found  in 
the  next  section.  Naturally,  these  papers  did  not  address  directly  the  objectives  of  reducing  the  operator- 
vehicle  ratio  and  optimising  interoperability.  Thus,  we  asked  selected  authors  to  comment  directly  on  the 
relationship  between  their  theoretical  framework  and  the  objectives. 

Two  options  were  discussed  at  the  Leiden  Workshop  that  could  help  streamline  and  focus  the  investigation  of 
theoretical  frameworks.  The  first  option  was  to  evaluate  frameworks  against  a  common  military  scenario 
(see  the  scenario  section,  Chapter  2,  para  2.1.6).  However,  if  there  is  a  significant  mismatch  between  theory 
and  operation  realities  then  the  comparison  may  be  invalid.  That  is,  elements  of  theoretical  framework  must 
be  related  to  elements  of  military  operations.  Also,  deciding  on  the  rules  for  adopting  frameworks  would  be  a 
formidable  task. 

The  second  option  (the  option  we  adopted)  was  to  take  a  closer  look  at  six  frameworks  and  ask  how  would 
they  address  operator-vehicle  ratio  and  interoperability  directly.  The  frameworks  could  be  compared  to 
standard  systems  design  approaches  and  to  each  other.  In  order  to  converge  onto  a  core  set  of  theoretical 
frameworks,  we  solicited  comments  from  selected  authors  and  asked  them  to  discuss  their  theory  or  model 
specifically  in  terms  of  optimising  the  operator- vehicle  ratio  and  interoperability.  This  option  is  limited  by  this 
Task  Group’s  limited  knowledge  and  reach  in  the  member  nations,  although  an  effort  was  made  to  be  as 
thorough  as  possible. 

At  the  Leiden  Workshop,  key  programs  and  individuals  who  contribute  to  UMV  human  factors  from  NATO 
countries  were  identified.  Table  3-1  provides  the  survey  questions,  which  were  developed  directly  from  the 
Terms  of  Reference  for  this  Task  Group.  The  survey  was  sent  to  10  individuals,  and  6  surveys  were  returned. 
Their  responses  are  found  in  Appendices  2  through  7. 

Table  3-1:  Survey  Questions 


Force  Multiplication 

•  Does  the  framework/model  address  operator  to  UMV  ratio  issues? 

•  If  so,  how  could  the  framework/model  help  reduce  the  ratio? 

•  Does  the  framework/model  address  interoperability  issues? 

•  If  so,  how  could  the  framework/model  help  improve  interoperability? 


UMV  Scenarios/Use-Cases 

•  Is  the  theory  applicable  to  UMV  situations  (i.e.,  underwater,  sea,  land,  air,  space)? 

Theory  Evaluation 

•  Has  the  framework/model  been  evaluated,  tested,  and  applied  to  commercial  or  military 
operations? 

•  Do  you  have  an  example,  closely  related  to  the  UMV  situations,  where  the  theory  was 
implemented? 
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3.2  FRAMEWORK  DESCRIPTIONS 

3.2.1  Literature  Review 

A  literature  survey  was  commissioned  to  assess  the  scope  (breath  and  depth)  of  the  literature  related  to 
operator-agent  interaction  for  intelligent  adaptive  interface  design.  The  actual  searching  fields  were  beyond 
suggested  areas  such  as  tele-robotics  and  human  computer  interaction,  and  included  supervisory  control, 
information  management  and  decision  support,  automatic  manufacturing,  medical  diagnosis  and  consultation, 
and  other  social  behaviour  areas.  More  than  500  abstracts  were  retrieved,  127  papers  and  reports  were  found 
likely  relevant  to  the  research  domain,  and  82  were  reviewed  of  which  27  dealt  with  theoretical  models  and 
empirical  evaluations.  A  detailed  review  was  made  for  seven  of  those  papers  and  these  brief  summaries  are 
listed  below. 


3.2. 1.1  Models  for  the  Design  of  Human  Interaction  with  Complex  Dynamic  Systems 
(Systems  Engineering  &  Management  Handbook,  1999) 

by  Mitchell,  C.M.  (1996) 

This  paper  provides  an  overview  of  the  evolution  of  human-machine  systems  models  over  the  past  forty  years. 
From  perspectives  of  cognitive  engineering,  ecological  psychology,  and  naturalistic  decision-making, 
the  similarities  between  human-machine  systems  models  and  a  variety  of  other  recent  approaches  to 
understanding  and  aiding  human  interaction  in  real-world  systems  were  described.  The  paper  also  proposed  a 
set  of  tenets  that  characterizes  models  and  human  interfaces  whose  design  was  based  upon  the  models. 

The  paper  described  the  trend  of  developing  robust  models  with  the  same  levels  of  fidelity  as  system  models. 
The  vision  was  to  predict  both  system  and  human  behaviour  and  provide  quantitative  assessments  of  the 
proposed  system  design.  In  60’s  and  70’s,  the  crossover  control  model  and  the  optimal  control  model 
(Wickens,  1984)  focused  on  continuously  tracking  behaviour  for  fully  manual  tasks.  However,  with  Digital 
Computer  introduced,  tasks  are  supervisory  controlling  of  multiple  subsystems.  Since  then,  modelling  objectives 
changed  from  the  pursuit  of  a  global  and  analytic/computational  operator  simulation  (i.e.,  a  quantitative, 
predictive  model)  to  a  more  focused  development  of  system  and  task  representations  that  could  be  used  for  the 
design  of  operator  interfaces  to  complex  dynamic  systems,  including  displays,  aids,  and  training  systems. 
Therefore,  the  objective  was  no  longer  to  produce  a  black-box  human  operator  simulator  that  functions  as 
robustly  as  traditional  engineering  models  of  system  hardware  and  software,  but  rather  the  development  of  a 
useful  description/prescription  of  the  system-task-operator  interactions.  Understanding  and  modeling  human 
cognition,  problem  solving,  and  decision-making  became  more  and  more  popular  and  practical  for  the  system 
design. 

The  essence  of  this  paper  is  the  emphasis  of  the  importance  of  context  as  the  central  tenet  of  Human-Machine 
Systems  Engineering.  As  Baron  (1984)  summarized  this  requirement  succinctly:  “[A  human-machine  systems 
model]  . . .  embodies  the  idea  that  to  model  human  performance,  one  must  model  the  system  in  which  that 
performance  is  embedded.”  Human  behaviour,  either  cognitive  or  psychomotor,  is  too  diverse  to  model  unless 
it  is  sufficiently  constrained  by  the  situation  or  environment;  however,  when  these  environmental  constraints 
exist,  to  model  behaviour  adequately,  one  must  include  a  model  for  that  environment  (the  perspective  of 
ecological  interface  design).  Therefore,  the  system  design  should  be  context-oriented,  descriptive,  and 
prescriptive. 
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3.2. 1.2  A  Theoretical  Analysis  and  Preliminary  Investigation  of  Dynamically  Adaptive  Interfaces 

(Journal  of  Aviation  Psychology,  11(2),  169-195,  Copyright  ©  2001) 

by  Bennett,  K.B.,  et  al.  (2001) 

This  paper  described  a  dynamically  adaptive  interface  (DAI),  which  changes  display  or  control  characteristics 
of  a  system  (or  both)  in  real  time.  The  goal  of  this  DAI  is  to  anticipate  informational  needs  for  desires  of  the 
users  and  provide  that  information  without  the  requirement  of  an  explicit  control  input  by  the  user. 
This  research  found  that  DAIs  have  the  potential  to  improve  overall  human-machine  system  performance  if 
they  are  properly  designed.  However,  DAIs  also  have  a  very  real  potential  to  degrade  performance  if  they  not 
properly  designed.  This  study  explores  both  theoretical  and  practical  issues  in  the  design  of  DAIs.  The  relation 
of  the  DAI  concept  to  decision  aiding  and  automation  was  discussed,  and  a  theoretical  framework  for  design 
was  also  outlined.  This  paper  shows  that  one  can  apply  theory  to  design! 

In  this  paper,  a  preliminary  investigation  of  the  DAI  design  concept  was  conducted  in  the  domain  of  aviation 
(precision,  low-level  navigation).  Three  interfaces  were  evaluated  including:  non-traditional  controls  (a  force 
reflecting  stick)  and  displays  (a  configurable  flight  director)  were  developed  to  support  performance  at  the 
task;  a  standard  interfaces  (conventional  controls  and  displays),  a  candidate  interface  (alternative  controls  and 
displays);  and  an  adaptive  interface  (dynamically  between  the  standard  and  candidate  displays).  The  results 
indicated  that  significant  performance  advantages  in  the  quality  of  route  navigation  were  obtained  with  the 
candidate  and  adaptive  interfaces  relative  to  the  standard  interface;  no  significant  differences  between  the 
candidate  and  adaptive  interfaces  were  obtained.  The  implication  of  these  results  was  discussed,  with  special 
emphasis  on  their  relation  to  fundamental  challenges  that  must  be  met  for  the  DAI  concept  to  be  a  viable 
design  alternative. 

This  paper  is  a  good  example  for  human-machine  interaction  from  theoretical  development  to  empirical 
investigation  to  maximize  overall  performance.  The  design  and  comparison  of  three  interfaces  is  a  typical 
method  for  such  kind  of  research.  The  results  are  also  very  interesting  as  they  raised  the  issue  of  the  dilemma 
for  automation  and  adaptation.  When,  where,  what,  why,  how  to  adapt  is  really  a  question  theoretically  and 
practically  for  better  operator-machine,  operator-agent  interaction.  The  theoretical  framework  and  research 
methodology  are  very  useful  for  other  similar  research. 


3.2. 1.3  Integrating  Perceptual  and  Cognitive  Modeling  for  Adaptive  and  Intelligent 
Human-Computer  Interaction 

(Proceedings  of  the  IEEE  Volume  90,  Issue  7,  July  2002,  Page(s):  1272-1289) 

by  Duric,  Z.,  et  al.  (2002) 

Through  both  theoretical  analysis  and  empirical  investigations,  this  paper  used  human  cognitive,  perceptual, 
motor,  and  affective  factors  to  adapt  the  interface  design  for  intelligent  human-computer  interaction. 
The  essence  of  the  paper  was  to  monitor  affective  behaviour  or  emotional  behaviour  or  non-verbal 
information  to  answer  what  why,  where,  when,  and  how,  and  then  adapt  the  display  according  to  this 
information.  The  method  for  interface  design  is  more  human-like  in  which  the  interface/machine  or  the  agent/ 
automation  embedded  is  regarded  as  another  human  assistant  who  can  monitor  perceptual  and  cognitive  levels 
and  understand  the  “partner”  or  “teammate”,  thus  react  for  better  collaboration  with  better  overall  results. 
The  idea  was  to  emphasise  the  team  collaboration  that  benefits  human-human  interaction.  “It  is  not  only 
computer  technology  that  needs  to  change  to  make  such  novel  interfaces  a  reality.  People  have  to  change  as 
well  and  adapt  to  the  interface  that  the  computer  presents  them  with.  In  the  end,  both  people  and  the  computer 
have  to  understand  each  other’s  intentions  and/or  motivations,  provide  feedback  to  each  other  as  necessary, 
and  eventually  adapt  to  each  other.” 
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3.2. 1.4  Adaptive  Interfaces  for  Human-Computer  Interaction:  A  Colourful  Spectrum  of 

Present  and  Future  Options 
(Proceedings  of  the  IEEE  1995,  292-297) 

by  Lajos,  B.  (1995) 

This  paper  discussed  adaptations  that  could  be  built  into  interfaces  for  human-computer  interaction  based  on 
different  aspects  of  human  behaviour  such  as  physiological  attributes  (eye,  ears,  fingers,  etc.),  intellectual 
characteristics  (capacity,  recognition,  learning,  decision,  etc.),  knowledge  basis  (knowing  the  environment, 
the  system,  him/herself,  etc.)  and  psychological  states  (concentration,  vigilance,  fatigue,  patience,  etc.). 
As  pointed  out  in  the  paper,  that  adaptive  interfaces  should  be  capable  of  adjusting  to  different  forms  of 
information  transfer,  transforming  the  information  contents,  altering/merging  modes  of  information  flow,  and 
exchanging/combining  communication  media. 

The  paper  also  discussed  the  future  of  adaptive  interfaces,  the  role  of  formal  interaction  modeling, 
the  importance  of  abstract/structural  interface  hierarchy,  the  integration  of  interaction  modes  and  media, 
the  sophistication  of  interface  modularity  and  the  exploitation  of  the  advantages  in  combining/integrating 
conceptual-functional-physical  design  aspects.  It  is  a  good  reference  for  the  taxonomy  of  adaptive  interfaces. 


3.2. 1.5  The  Future  of  Watchstation  Design:  Evolution  from  Single  Purpose  to  Intelligent 

Watchstations 

(2002  Command  and  Control  Research  and  Technology  Symposium, 

Naval  Postgraduate  School,  Monterey,  California,  11-13  June  2002) 

by  O’Donnell,  L.  (2002) 

The  focus  of  this  paper  is  to  address  the  issues  in  interface  design  and  the  changes  in  the  console  design  to 
support  distributed  mission  task  activities  for  joint  operations  of  global  command  and  control  systems. 
Increased  mission  demands  combined  with  smith  weapons,  automated  functions  and  increased  collaborative 
warfighter  functions  have  increased  the  multi-tasking  requirements  to  be  accomplished.  Humans  in  a 
warfighter  role  have  shifted  from  a  narrow  task  focus  within  a  narrow  job  focus  of  a  single  purpose 
watchstation  and  a  high  human-in-the  loop  interface  workload,  to  becoming  controllers  of  these  distributed 
systems  and  collaborative  activities.  In  the  paper,  current  watchstation  design  was  described  that  requires  the 
human  to  perform  manual  system  operations  in  combination  with  numerous  independent  synchronous 
activities  such  as  communications  and  adjacent  equipment  operations.  Future  watchstation  will  need  to  be 
designed  to  support  the  work  environment  with:  increased  multi-tasking  capabilities,  dynamic  monitoring  of 
task  process,  integrated  system  designs,  and  improved  distributed  team  collaboration  task  capabilities. 
Advances  in  technology  have  enabled  the  design  of  an  effective  watchstation  design  that  will  allow  for  multi¬ 
modal  user  interfaces  best  suited  to  the  task.  Future  watchstation  designs  should  also  utilize  self-adaptive 
interfaces,  increased  visual  workspaces,  agent  technologies,  integrated  speech,  and  visual  and  direct  touch 
methods  to  reduce  the  human-interface  workload  and  streamline  the  tasks.  All  these  features  are  required  for 
future  UAV/UCAV  control  interfaces. 

The  paper  not  only  analyzed  the  current  trends  and  advantages  of  intelligent  interfaces  (watchstations), 
but  also  brought  up  a  smart  agent  taxonomy  to  construct  the  flexible,  dynamic,  scalable,  and  robust  distributed 
system  capabilities  over  system  networks  as  multiple  agent  systems.  Although  the  context  discussed  in  the 
paper  was  not  focused  on  airborne  multiple  UAV  control,  but  the  discussions  of  several  key  technologies  to 
enabling  an  intelligent  system  and  future  research  recommendations  on  intelligent  user  interface  design  can  be 
generally  applied  to  any  design  of  the  framework  for  optimal  operator  -  agent  interaction. 
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3.2.1.6  An  Architecture  for  Intelligent  Interfaces:  Outline  of  an  Approach  to  Supporting 
Operators  of  Complex  Systems 
(Human  Computer  Interactions,  3(2),  87-122) 

by  Rouse,  W.B.,  Geddes,  N.D.  and  Curry,  R.E.  (1997-1998) 

This  paper  described  a  concept  of  a  comprehensive  support  system  design  for  operators  of  complex  systems. 
A  variety  of  difficult  design  issues  were  addressed  as  well  as  ongoing  efforts  aimed  at  resolving  these  issues. 

The  main  focus  of  the  paper  was  to  address  design  methodology  and  automation  philosophies.  Although  the 
suggested  design  methodology  follows  the  traditional  human  factors  engineering  principles,  automation 
philosophy  emphasises  on  maximizing  overall  performance  by  overcoming  human  limitations  and  enhancing 
human  abilities.  The  philosophy  is  that  automation  should  be  used  as  a  backup  -  the  default  modes  are  usually 
manual  with  automation  invoked  only  when  either  anticipated  operator  performance  is  unacceptable  or  the 
operator  chooses  to  relinquish  control.  With  the  adoption  of  this  operator-centred  automation  philosophy, 
an  architecture  including  intelligent  management  of  information  and  tasks  was  proposed.  Within  it, 
the  concept  of  operator  state  is  central  to  the  functioning  of  the  components  of  the  intelligent  interface. 
The  relevant  elements  include:  activities,  awareness,  intentions,  resources,  performance.  Another  component 
is  the  interface  manager,  which  is  similar  to  an  executive’s  assistant  who  zealously  guards  the  superiors’  time 
and  resources.  Although  important  questions  were  raised  about  what,  when,  and  how  to  automate,  there  was 
no  clear  answer  in  the  paper.  The  paper  presented  a  layout  of  the  architecture  for  complex  system  design, 
but  it  did  not  cover  enough  cognitive  and  perceptual  aspects  of  a  dynamic,  complex,  and  interactive  system, 
especially  for  multiple  UMV  control. 


3.2. 1.7  A  Model  for  Types  and  Levels  of  Human  Interaction  with  Automation 
(IEEE  Transactions  on  Systems,  Man  and  Cybernetics,  30  (3),  286-297) 

by  Parasuraman,  R.,  Sheridan,  T.B.  and  Wickens,  C.D.  (2000) 

A  framework/model  was  proposed  for  types  and  levels  of  human  interaction  with  automation  (Sheridan’s 
10  points  scale).  They  also  proposed  four  broad  classes  of  functions  which  automation  could  be  applied  from 
information  processing  point  of  view:  a)  information  acquisition,  b)  information  analysis,  c)  decision  and 
action  selection,  and  d)  action  implementation.  Within  each  of  these  types,  automation  can  be  applied  across  a 
continuum  of  levels  from  low  to  high,  from  fully  manual  to  fully  automatic.  A  particular  system  can  involve 
automation  of  all  four  types  at  different  levels.  Since  automation  does  not  merely  supplant  but  changes  human 
activity  and  can  impose  new  coordination  demands  on  the  operator,  appropriate  selection  is  important  based 
on  the  primary  and  secondary  evaluative  criteria.  The  primary  criteria  look  at  human  performance 
consequences:  mental  workload,  situation  awareness,  complacency,  and  skill  degradation.  The  second  criteria 
include  the  automation  reliability  and  the  costs  of  consequences. 

Although  the  paper  considered  human  machine  interaction  mainly  from  information  process  point  of  view,  the 
proposed  model  could  be  a  good  starting  point  for  considering  what  types  and  levels  of  automation  should  be 
implemented  in  a  particular  system.  The  paper  is  concerned  with  human  performance  in  automated  systems 
and  emphasizes  human-machine  comparison.  Automation  is  defined  as  a  device  or  system  that  accomplishes 
(partially  or  fully)  a  function  that  was  previously,  or  conceivably  could  be,  carried  out  (partially  or  fully)  by  a 
human  operator.  The  paper  also  touched  a  little  bit  on  action  automation  -  agents  which  track  user  interaction 
with  a  computer  and  execute  certain  sub-tasks  automatically  in  a  contextually-appropriate  manner.  However, 
this  area  should  be  elaborated  more  associated  the  theory  developments  and  empirical  investigations  (which  is 
the  future  work  as  mentioned  in  the  paper).  Even  in  the  proposed  model,  the  issue  in  whether  automation 
unreliability  has  similar  negative  effects  for  all  four  stages  of  automation  needs  further  examination. 
Regarding  costs  of  decision/action  outcomes,  individual  differences  between  users  in  the  same  interface 
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should  be  addressed  more,  especially  on  user  profile  building  on  the  interface  (user  modelling  embedded  in 
interface).  It  is  good  to  point  out  that  empirical  work  needs  to  be  done  to  explicitly  compare  the  effects  on  human 
performance  of  different  levels  of  automation  for  information  acquisition  and  analysis,  in  other  words,  the  levels 
of  interface  intelligence.  Overall,  this  paper  emphasised  more  function  allocation  between  user  and  machine 
regarding  automation,  but  less  was  discussed  on  operator  interface  interaction.  How  the  automation  should 
perceive,  analyze,  understand,  react,  and  collaborate  with  user  as  an  agent  or  assistant  still  remains  untouched. 

3.2.2  Frameworks  from  Survey  Returns 

The  following  framework  descriptions  were  those  of  the  six  authors  who  responded  to  the  survey.  They  serve 
as  the  main  frameworks  from  which  we  can  find  any  common  or  unique  theoretical  elements  required  to 
analyse  UMV  systems. 

3.2.2.1  Cognitive  Automation  (CA) 

Cognitive  Automation  makes  a  distinction  between  conventional  automation  and  cognitive  automation  [10]. 
Also  see  chapter  5  for  a  full  description  of  the  theory.  Conventional  automation  is  predominantly  focused  on 
subgoals  and  subtasks  of  the  work  process.  High-level  goals  and  their  associated  tasks  are  not  known  by  the 
automation  system.  Consequently,  these  systems  observe  only  a  small  portion  of  what  is  relevant  in  a  given 
situation.  For  example,  if  the  “altitude  hold”  autopilot  is  activated,  it  is  doing  its  best  to  comply  with  this 
assigned  reference  or  goal  state,  even  if  a  high  mountain  is  in  its  way.  The  autopilot  does  not  know,  nor  care 
about  the  top-level  safety  goal  of  avoiding  a  crash  into  the  mountain  -  this  goal  is  exclusively  in  the  realm  of 
the  high-level  operating  element  responsibilities  (usually  human  supplemented  by  a  machine).  Never  the  less, 
conventional  automation  is  very  convenient  for  simple  automated  functions  where  the  supervisor  has  direct 
visibility  and  influence  on  the  states  that  the  automation  has  control  over. 

Cognitive  automation  takes  into  consideration  these  high-level  goals.  The  authors  present  the  idea  of  an 
Artificial  Cognitive  Unit  (ACU)  that  implements  a  model  of  the  Cognitive  Process,  and  the  ACU  has  been  coded 
in  software.  The  ACU  includes  processing  steps  of  interpretation  ending  up  with  beliefs,  goal  determination, 
planning,  and  plan  realisation  to  generate  the  instructions  for  action.  These  processing  steps  are  based  upon  a 
priori  knowledge. 

ACUs  have  the  potential  to  achieve  high-level  goals  compliant  with  those  of  the  human  operator,  and  therefore 
may  act  as  an  operator  assistant  system  as  well  as  act  autonomously.  Then,  a  “cognitive”  autopilot  will  take  into 
consideration  the  mountain  in  front,  will  know  that  to  proceed  stubbornly  with  altitude  hold  will  end  in  disaster 
and  it  will  look  for  a  way  around  all  based  upon  the  high-level  goal  for  safe  flight. 

3.2.2.2  Extended  Control  Model  (ECOM) 

The  Extended  Control  Model  (ECOM)  has  been  described  in  a  conference  paper  [11]  and  a  book  chapter  in  a 
textbook  [12].  The  pedigree  of  the  model  includes  the  description  of  the  principle  of  contextual  control  [13], 
the  initial  contextual  control  model  [14],  and  the  fully  developed  contextual  control  model  [15].  Also  see 
chapter  7  for  a  full  description  of  the  theory. 

Briefly  explained,  the  model  provides  a  framework  for  describing  how  a  joint  cognitive  system  can  maintain 
control  of  a  situation  or  a  process.  A  cognitive  system  is  defined  as  “a  system  that  can  modify  its  behaviour  on 
the  basis  of  past  experience  so  as  to  achieve  specific  antientropic  ends.”  A  joint  cognitive  system  can  be  any 
combination  of  humans  and  machines  (technological  artifacts)  or  humans  and  humans  (social  groups/ 
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organisations).  The  model  invokes  the  principle  of  multiple,  simultaneous  layers  of  control.  The  layers  are 
hierarchically  organised  with  one  or  more  instances  at  each  layer.  Control  layers  differ  with  respect  to  the  time 
window  or  time  horizon  they  cover,  as  well  as  in  the  balance  between  feedback  and  feedforward  control. 
The  model  currently  describes  four  layers  of  control  called  (from  the  top  down)  targeting,  monitoring, 
regulating,  and  tracking.  It  is  applicable  both  to  single  systems  (e.g.,  a  driver  and  a  vehicle)  and  to  larger 
entities  and  organizations. 

3.2.2.3  Multiple  Agent  Interaction  (MAI)  Model 

A  framework  is  proposed  based  on  Perceptual  Control  Theory  (PCT)  [16]  to  model  multiple  non-interacting 
and  interacting  agents  [17].  Agents,  in  this  case,  may  be  human  or  software  agents  that  mimic  human 
cognition  and  behaviour.  This  modelling  activity  was  performed  in  order  to  understand  the  system  dynamics 
of  human-intelligent  agent  interaction,  which  would  lead  to  design  implications.  PCT  describes  human 
cognition  as  a  means-end  hierarchical  network  of  control  units.  Each  control  unit  involves  the  control  of  a 
perception.  At  the  lowest  levels,  control  is  subconscious  and  might  be  described  by  classical  linear  control 
theory.  At  the  highest  levels,  control  is  conscious  and  deliberate  requiring  rule-based  thinking,  logic, 
and  reasoning.  Regardless  of  the  hierarchical  level,  the  control  law  remains  the  same  -  the  output  of  each 
control  loop  attempts  to  drive  the  perception  closer  to  its  internal  goal.  The  key  advantage  of  modelling  agents 
using  PCT  is  that  one  can  apply  all  the  mathematical  power  of  control  theory  including  stability  and 
optimization  analyses. 

From  a  control  theory  perspective,  human-machine  interaction  is  often  analysed  by  treating  the  machine  as  a 
simple  input-output  transfer  function  with  some  known  disturbances,  and  the  human  is  part  of  the  machine’s 
controller  algorithm  [18].  On  the  other  hand,  multiple  agent  interaction  should  be  treated  as  separate  control 
loops  that  have  independent  reference  values,  and  that  interact  only  through  the  influencing  and  sensing  of 
common  states  of  the  physical  world.  Conceptually,  this  interaction  model  would  produce  a  complex  set  of 
dynamics  that  are  not  always  stable  and  not  easily  predictable.  Thus,  as  the  operator-vehicle  ratio  is  reduced 
by  adding  more  intelligent  agents,  there  is  a  risk  of  producing  an  unstable  system  if  not  carefully  designed. 

The  following  principles  were  highlighted  during  the  analysis  of  multiple  agent  interaction  using  control 
theory  techniques: 

•  Closed-loop  feedback  modelling  techniques  can  be  used  in  the  design  of  multiple  agent  systems. 

•  Designers  should  consider  goals,  sensing  and  decision-making  strategies,  and  world  states  as  part  of 
their  system  design. 

•  Agents  should  act  on  separate  states,  while  gathering  data  from  all  sources. 

These  design  principles  have  been  applied  to  UMV  research  studies  on  the  design  of  intelligent  adaptive 
interfaces,  and  selecting  crews  for  UAV  operations.  The  third  design  philosophy  was  applied  to  information 
management  business  rules  for  the  Multi-National  Experiment  4  on  Effects  Based  Operations  with  US  Joint 
Forces  Command  as  the  experiment  lead.  This  experiment  involved  over  one  hundred  players  over  a 
distributed  network  from  countries  around  the  world.  The  interface  design  and  business  rules  were  critical  in 
order  to  successfully  collaborate  and  conduct  the  experimental  operation. 

3.2.2.4  Military  Relevance  Philosophy  (MRP) 

See  Chapter  2  for  a  full  description  of  the  theory.  The  human  axiom  is  to  facilitate  the  development  of 
technology  and  the  use  of  it  to  actually  serve  the  human  in  the  best  possible  way.  With  respect  to  UMVs, 
the  human  axiom  is  to  put  the  technologies  of  automation  and  uninhabited  systems  into  a  necessary  context  in 
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order  to  reduce  the  risk  of  having  these  capabilities  become  counterproductive.  This  is  especially  important 
when  it  comes  to  highly  automated  and  uninhabited  systems.  Military  effect  is  nothing  but  humans 
influencing  other  humans  using  military  powers,  commonly  facilitated  by  technological  systems.  The  axiom  is 
to  state  that  if  there  is  no  military  human  completely  in  charge  of  the  systems’  effect,  then  the  effect  isn’t 
actually  military  and  thus  has  no  military  relevance. 

By  nature,  it  is  impossible  to  exactly  predict  the  future  since  every  situation  contains  a  certain  amount  of 
uncertainty,  which  here  is  defined  as  the  concept  of  Situation  Uncertainty  (SU).  Furthermore,  every  situation 
changes  continuously,  and  increases  with  amount  of  time  before  an  event.  Having  control  or  being  in  control 
of  situations  with  very  high  uncertainty  is  difficult.  It  is  a  significant  difference  between  to  have  control  and  to 
perform  control.  The  state  of  having  control  is  a  possible  result  of  the  work  done  by  performing  control,  but  it 
is  not  a  necessary  consequence  of  control  (that  is,  even  though  control  has  been  applied  a  system  may  diverge 
because  of  high  uncertainty).  To  be  able  to  have  control  there  must  be  awareness  about  the  situation 
(i.e.,  reduced  uncertainty),  the  entity’s  ability  to  control  its  environment,  and  the  control  of  the  entity  itself. 

When  it  comes  to  “controlling  humans”,  the  concept  of  trust  may  replace  the  concept  of  control,  because 
interacting  entities  are  autonomous,  self-aware  agents.  Humans  deal  with  uncertain  and  complex  situations 
trusting  that  the  other  human  team  member  will  act  according  to  common  values.  One  can  imagine  that  trust 
will  play  a  significant  role  amongst  human  operators  and  autonomous  UMV  systems.  Humans  may  grow  to 
trust  the  machine,  but  trust  must  be  mutual  under  uncertain  situations.  The  technological  system  must  not  only 
trust  the  human,  but  also  itself  particularly  when  the  system  functions  outside  its  design  envelope  (i.e.,  SU). 
This  is  especially  true  for  uncertainties  in  the  higher  levels  of  control,  uncertainties  about  what  to  do  and  why 
it  has  to  be  done,  and  less  true  when  it  comes  to  uncertainties  about  how  to  do  things.  Current  technologies  do 
not  have  the  capability  to  trust  as  defined  herein. 

Which  results  in  the  human  axiom: 

“Every  reduction  of  human  control  over  technology  is  negative.  If  control,  in  defiance  of  that,  is 
to  be  reduced  it  must  be  completely  justified.  Reduction  of  control  may  be  justified  if  and  only  if 
the  worst  possible  consequences  of  the  reduced  control  are  exceeded  by  the  benefits  of  reducing 
the  control  (Patrik  Stensson).  ” 

Relative  strengths  (i.e.,  safety  and  force  multiplication)  follow  directly  from  separation  of  human  and 
platform.  Relative  weaknesses  are  mostly  consequences  of  reduced  control  because  of  the  same  separation  as 
well  as  the  lack  of  mutual  trust.  Relevance  of  technological  solutions,  especially  military  ones  that  might 
deliberately  cause  harm,  needs  to  be  judged  thoroughly. 

3.2.2.5  Playbook  (PB) 

As  Unmanned  Military  Vehicles  become  more  intelligent  and  capable,  and  as  there  is  an  attempt  to  control 
more  of  them  with  fewer  humans  in  the  loop,  there  is  a  need  to  move  toward  a  model  of  delegation  of  control 
rather  than  the  direct  control  that  characterizes  current  practice  [19].  Also  see  Chapter  7  for  a  full  description 
of  the  theory.  Five  delegation  methods  are  identified  and  described,  and  can  serve  as  building  blocks  from 
which  to  compose  complex  and  sensitive  delegation  systems:  delegation  through: 

1)  Providing  goals; 

2)  Providing  full  or  partial  plans; 

3)  Providing  negative  constraints; 
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4)  Providing  positive  constraints  or  stipulations;  and 

5)  Providing  priorities  or  value  statements  in  the  form  of  a  policy. 

The  Playbook  architecture  supports  delegation  action  types  1  -  4  in  principle  and  has  been  implemented  in 
prior  prototypes  to  include  action  types  2  and  4.  While  the  work  described  above  represents  a  general 
framework  for  delegation  interactions  suitable  for  human  interaction  with  smart  automation  of  various  kinds 
and,  perhaps  uniquely,  suitable  for  the  tasking  of  multiple  UMVs,  the  work  has  thus  far  progressed  only  to  the 
proof  of  concept  stage.  The  Playbook  ‘proper’  consists  of  a  User  Interface  (UI)  and  a  constraint  propagation 
planner  known  as  the  Mission  Analysis  Component  (MAC)  that  communicate  with  each  other  and  with  the 
operator  via  a  Shared  Task  Model.  The  operator  communicates  instructions  in  the  form  of  desired  goals,  tasks, 
partial  plans  or  constraints,  via  the  UI,  using  the  task  structures  of  the  shared  task  model.  The  MAC  is  an 
automated  planning  system  that  understands  these  instructions  and  (a)  evaluates  them  for  feasibility  and/or  (b) 
expands  them  to  produce  fully  executable  plans.  The  MAC  may  draw  on  special  purpose  planning  tools 
(e.g.,  an  optimizing  path  planner)  to  perform  these  functions,  wrapping  them  in  its  task-sensitive  environment. 
Outside  of  the  tasking  interface,  but  essential  to  its  use,  are  two  additional  components.  An  Event  Handling 
component,  itself  a  reactive  planning  system  capable  of  making  momentary  adjustments  during  execution, 
takes  plans  from  the  Playbook.  These  instructions  are  sent  to  control  algorithms  that  actually  effect  behaviors. 

The  final  type  of  delegation  interaction  offers  the  ability  to  provide  priorities  between  alternate  goals  and  states 
and  to  do  so  more  abstractly  than  the  above  methods.  These  abstract  value  statements  that  a  supervisor  might 
provide  are  referred  to  as  his  or  her  “policy”  for  performance  in  the  domain.  A  policy  statement  is  an  abstract, 
general,  a  priori  statement  of  the  relative  importance  or  value  of  a  goal  state  in  the  domain.  In  its  simplest  form, 
policy  provides  a  method  for  human  operators  to  mathematically  define  what  constitutes  “goodness.” 

The  work  described  above  represents  a  general  framework  for  delegation  interactions  suitable  for  human 
interaction  with  smart  automation  of  various  kinds.  This  work  is  in  the  proof  of  concept  stage  although 
exploration  of  Playbook-lilce  interfaces  is  being  conducted  (under  a  DARPA-IXO  SBIR  grant).  One  of  the 
goals  of  this  work  will  be  to  develop  task  libraries  and  task  construction  tools  and  interface  concepts  to  move 
the  delegation  interface  work  along  toward  implementation  and  utility. 

3.2.2.6  System  Process/Task  Organisational  Model  for  HF  V&V  (SPTO) 

If  “The  Human  Factors  of  Augmenting  the  Force”  is  indeed  the  objective  then  the  outcomes  produced  should 
have  enough  practicality  to  consider  their  applicability  in  augmenting  force  effectiveness  in  the  ‘real’  world. 
Amongst  other  considerations,  it  is  important  to  consider  the  verification  and  validation  (V&V)  of  a 
theoretical  framework  for  augmenting  the  force.  V&V  is  made  both  statically  (verification)  and  dynamically 
(validation)  -  statically  to  ensure  that  it  is  basically  logical  and  fit  for  intended  purpose,  dynamically  to 
examine  its  fit  to  reality  [20]. 

The  System  Process/Task  Organisational  Model  for  HF  V&V  presented  at  Leiden  was  shown  using  an 
embedded  Theoretical  Framework  of  Task  Organisation.  In  the  author’s  view  that  framework  always  implies 
levels  of  consideration  on  Task  Context,  Situation,  Mission,  Goal(s)  [possibly  both  Strategic  and  Tactical], 
Force  Structure,  Organisations,  Cultural  influences,  the  role(s)  of  technology,  required  system  functions  and 
performance,  roles  of  personnel,  jobs,  teamwork,  individual  tasks,  and  individual  actor  /  entity  properties  and 
activities. 

If  a  system  is  considered  dynamically  it  is  with  a  task  related  perspective;  if  statically  it  is  more  at  a  function, 
constraint,  architecture,  capability,  or  generic  process  perspective  of  consideration.  However,  the  model  is 
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seldom  expressed  explicitly.  The  model  relies  on  checks  on  the  effectiveness  of  the  progress  of  a  system 
through  its  set  process  phases  (i.e.,  What  has  occurred  and  When  by  forms  of  Measures  of  Effectiveness 
[MOEs]).  To  provide  explanation  of  the  quality  of  work  performance  throughout  the  phases  of  the  process  an 
associated  Task  Organisational  Model  provides  evidence  of  the  quality  of  work  performance  to  satisfy 
questions  on  How,  Why,  Where,  and  by  Whom  task  effort  has  been  applied  in  support  of  the  satisfaction  of 
the  goals  of  the  system  process  (here  possibly  by  the  use  of  forms  of  Measures  of  Performance  as  associated 
with  MOEs). 

If  proof  is  required  that  the  application  of  HF  to  UMV  life  cycle  issues  will  augment  the  force  then  it  is 
necessary  to  provide  a  theoretical  model  with  associated  measures  that  if  applied  in  practice  can  produce 
evidence  of  the  degree  that  the  application  of  HF  has  succeeded  in  its  aim.  One  such  model  has  been 
proposed.  It  is  suggested  that  regardless  of  the  form  or  efficacy  of  HF  practices  applied  to  the  life  cycle  issues 
of  employment  of  UMVs  in  a  force,  that  the  adoption  of  some  form  of  a  System  Process/Task  Organisational 
Model  is  necessary  to  fully  evaluate  a  system,  its  capability,  and  its  quality  of  fitness  for  purpose. 


3.3  SURVEY  RESULTS 

The  following  table  summarizes  the  survey  results.  For  brevity,  key  sentences  and/or  phrases  are  taken  from 
the  responses  in  Appendices  1  through  7.  For  more  detail  please  refer  to  the  Appendices. 
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Table  3-2:  Survey  Results 


Question 

CA 

ECOM 

MAI 

MRP 

PB 

SPTO 

Does  the  framework/model  address 
operator  to  UMV  ratio  issues? 

Yes. 

Yes. 

Yes. 

No. 

Yes. 

Yes. 

If  so,  how  could  the  framework/ 
model  help  reduce  the  ratio? 

Will  supplement 
human  teams  and 
thus  affect  operator 
to  UMV  ratio. 

Enables  a 
constructive 
discussion  of 
allocations. 

Has  the  potential  to 
show  optimal 
trajectories  through 
the  state  space  as  the 
ratio  is  reduced. 

Will  help  in 
increasing  the 
operator  to 

UMV  ratio 
instead  of 
reducing  it. 

Permits  a  single 
operator  to  do 
everything  from 
joystick  control  to 
swarm  control  of  any 
number  of  UAV s . 

Considers 
automation, 
autonomy, 
manning,  and 
personnel. 

Does  the  framework/model  address 
interoperability  issues? 

Yes. 

Yes,  Indirectly. 

Yes. 

Yes. 

Yes. 

Yes. 

If  so,  how  could  the 
framework/model  help  improve 
interoperability? 

Will  handle 
problems  covering 
the  full  scope  of 
interoperability 
levels  -  both  high 
and  low. 

Give  guidelines 
about  which 
information  is 
needed. 

Standards  might 
ensure  proper  sensing 
of  information,  and 
action  must  be 
coordinated  to  ensure 
de-confliction. 

Appropriate 
human  control 
is  necessary  to 
achieve 

interoperability. 

PB  achieves 
interoperability  by 
building  the  knowledge 
about  how  to  utilize 
different  resources 
into  PB’s  Planner. 

Task 

organizational 
model  can  be 
developed  to 
consider 
interoperability. 

Is  the  theory  applicable  to  UMV 
situations  (i.e.,  underwater,  sea, 
land,  air,  space)? 

Yes. 

Yes  (implied). 

Yes. 

Yes. 

Yes  (implied). 

Yes,  Indirectly. 

Has  the  framework/model  been 
evaluated,  tested,  and  applied  to 
commercial  or  military  operations? 

Yes. 

Yes. 

Yes. 

Not  applicable. 

Yes. 

Yes,  Implicitly. 

Do  you  have  an  example,  closely 
related  to  the  UMV  situations, 
where  the  theory  was  implemented? 

COSY  flight. 

COSA. 

Mini-UAV  field 
demonstration. 

Manned-unmanned 

teaming. 

Automobiles. 

Nuclear  Power 
plant. 

Closely  related 
UMV  example. 

Multiple  UAV 
interface  design. 

Not  Applicable. 

DARPA  UAV  project, 
PVACS. 

No,  only 
coalition 
operations. 
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3.4  DISCUSSION 

3.4.1  Operator  to  UMV  Ratio 

Five  out  of  six  authors  indicated  that  their  framework  would  contribute  to  reducing  the  operator- vehicle  ratio. 
This  was  to  be  expected  since  the  authors  were  invited  to  comment.  All  but  MRP  indicated  how  this  would  be 
accomplished,  or  at  the  very  least,  how  various  operator- vehicle  configurations  could  be  analysed. 

CA  suggested  that  the  Artificial  Cognitive  Unit  could  perform  some  high-level  cognitive  tasks  that  would  be 
otherwise  done  by  the  human,  thus  freeing  the  operator  to  manage  multiple  UMVs.  ECOM  provides  a 
theoretical  basis  to  discuss  levels  of  control  and  control  allocation.  MAI  would  caution  that  the  real  issue  is 
not  the  ratio  of  operators  to  vehicles,  but  that  the  addition  of  (human  or  machine)  intelligent  agents  increases 
the  chances  for  instabilities.  MRP  would  argue  that  if  humans  are  to  remain  in  control,  then  the  ratio  should  be 
increased  rather  than  reduced.  Given  that  the  operator  is  the  “coach”  and  the  UMVs  are  “players”  then  PB 
would  claim  that  there  could  be  any  number  of  UMVs  that  adhere  to  the  playbook.  SPTO  says  that  automation 
and  manning  should  be  considered  in  parallel. 

In  general,  frameworks  provide  a  common  lexicon  and/or  analogy  around  which  any  specific  configuration  may 
be  described  and  compared  to  other  configurations.  Some  type  of  framework  or  model  is  required  to  predict 
performance  when  the  operator-vehicle  ratio  is  reduced.  Without  this  modeling  investigation,  it  becomes 
increasingly  difficult  to  have  confidence  that  the  system  will  work. 

3.4.2  Interoperability 

Contributors  inferred  that  their  framework  addresses  interoperability,  although  there  is  some  suggestion  that 
interoperability  was  interpreted  differently  amongst  the  theories.  In  all  cases,  human  interoperability  at  the 
highest  level  of  abstraction  was  included  in  the  interpretation.  CA  claims  to  cover  the  full  scope  of 
interoperability.  ECOM  would  provide  guidelines  for  the  information  needs.  MAI  would  produce  standards 
for  information  exchange  between  agents.  MRP  focused  on  human  control  as  the  means  of  achieving 
interoperability.  PB  has  a  Planner  that  includes  the  use  of  resources.  SPTO  may  require  some  modifications  so 
that  the  model  addresses  interoperability. 

3.4.3  Applications  and  UMV  Situations 

The  survey  provided  three  questions  on  theory  applicability,  theory  application,  and  specific  UMV  situations. 
The  intent  was  not  to  exclude  a  theory  because  it  had  not  been  applied  to  a  UMV  situation,  yet  weight  those 
theories  that  have  been  applied  to  UMV  situations  highly.  All  theories  are  applicable  to  UMV  situations 
according  to  the  respondents.  Five  out  of  six  frameworks  have  been  tested  and  applied  to  commercial  or 
military  operations.  MRP  is  a  philosophy  and  is  strictly  applicable  to  these  questions.  From  the  list  of  UMV 
situations,  CA,  ECOM,  and  PB  seemed  to  be  the  most  mature  followed  by  MAI.  SPTO  has  been  applied  to 
coalition  operations,  and  MRP,  again,  is  not  applicable. 

Like  any  application  of  theory  there  are  compromises  usually  between  cost,  effort,  and  time.  The  final  design 
is  usually  a  “healthy”  compromise  between  the  theory  and  practical  considerations.  The  key  point  here  is  that 
frameworks  have  gone  from  theory  to  implementation  with  positive  and  useful  results. 
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3.4.4  Common  and  Unique  Framework  Elements 

This  section  provides  a  first  look  at  some  of  the  elements  that  all  the  frameworks  have  in  common,  and  then 
highlights  some  of  the  unique  framework  elements.  The  first  common  element  is  the  notion  of  control  theory. 
That  is,  most  theories  have  the  notion  of  desired  states  and  actual  states  in  conjunction  with  sensing  the  world 
and  influencing  world  states.  This  seems  to  be  true  for  animate  or  inanimate  actors  within  the  environment. 

Another  common  element  is  the  idea  of  a  hierarchy.  There  is  an  admission  that  the  framework  should  be  able 
to  cope  with  multiple  levels  of  abstraction  -  from  data  sensing  and  perception  through  to  higher  goal  setting, 
reasoning,  and  decision-making.  If  the  model  is  a  single  level  abstraction,  then  it  is  might  not  be  able  to 
address  operator-UMV  interaction. 

A  third  common  element  is  that  most  frameworks  advocate  the  need  for  the  actor  to  sense  the  world 
(environment),  sense  its  own  state,  sense  the  state  of  other  actors  within  its  immediate  environment  (including 
the  team  lead),  and  understand  the  mission  objectives.  Effectively,  the  algorithm(s)  must  at  least  mimic  to 
some  extent  human  cognition  what  humans  seem  to  naturally  do  when  working  in  a  team  of  humans 
(see  Chapter  7). 

Descriptive  analyses  are  another  common  element  amongst  most  frameworks.  That  is,  investigation  of 
operator- vehicle  ratio  or  interoperability  is  done  primarily  by  logically  reasoning  through  the  topics  using  the 
theory  as  a  frame  of  reference.  Some  of  the  more  mature  theories  have  shown  the  instantiation  of  their 
framework  in  a  product  or  by  modelling  and  simulation  to  show  that  the  concepts  indeed  work. 

A  final  common  element  is  that  the  theory  often  leads  to  design  philosophy  and  guidelines.  In  the  case  of 
MRP  and  SPTO,  the  design  considerations  begin  with  philosophical  statements.  It  becomes  difficult  to 
separate  the  model  from  the  model’s  underpinning  philosophy  and  perhaps  it  is  not  necessary  to  make  this 
distinction.  Only  that  a  practical  output  of  theory  and  philosophy  are  design  guidelines. 

One  of  the  unique  framework  elements  includes  PB  with  the  notion  of  a  playbook.  This  is  not  precisely  rule- 
based  decision-making,  nor  is  it  free  play.  It  does  pre-suppose  cooperative  actors  working  together  to  achieve 
a  common  objective.  Not  every  actor  knows  precisely  what  the  other  actor  is  doing,  but  at  least  their  actions 
are  consistent  and  they  are  moving  toward  a  common  goal.  This  uniqueness  has  tremendous  potential  for  an 
operator  managing  multiple  UMVs. 

Another  unique  framework  element  is  the  mathematical  analysis  of  MAI.  That  is,  symbolic  mathematics  was 
used  to  draw  conclusions  from  the  framework.  On  the  other  hand,  the  assumptions  that  were  applied  to  make 
the  mathematics  tractable  could  be  easily  argued  that  the  resultant  model  does  not  reflect  the  true  situation. 
Never  the  less,  experimental  results  have  shown  that  the  theory  has  some  validity  and  that  the  mathematics  is 
only  an  alternative  to  descriptive  analyses  to  come  to  similar  conclusions  about  multiple  agent  interaction. 

A  final  unique  element  is  the  notion  that  a  reduction  in  the  operator-vehicle  ratio  is  likely  to  lead  to  an 
increase  in  situation  uncertainty  as  stated  by  MPR.  This  seems  like  a  logical  statement,  but  one  that  designers 
have  a  tendency  to  forget.  Control  theory  has  the  notion  of  controllability  and  observability.  That  is,  a  stable 
system  has  both  observable  states  (states  that  can  be  sensed  directly)  that  are  controllable  (can  be  influenced 
directly).  However,  states  that  can  neither  be  influenced  nor  sensed  directly  have  a  greater  chance  of  being 
unstable  (i.e.,  increased  uncertainty).  The  same  principle  may  happen  with  a  human  operator  with  limited 
capacity  to  sense  and  influence  all  of  the  UMVs  simultaneously.  There  are  just  too  many  variables  to  track. 
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3.5  SUMMARY 

The  purpose  of  this  chapter  is  to  inform  the  reader  for  the  need  to  consider  a  theoretical  framework  throughout 
the  life  cycle  of  UMVs.  The  theoretical  framework  helps  bound  the  problem  and  provides  guidance  on  how  to 
operate  the  new  system.  A  large  number  of  theoretical  frameworks  were  reviewed  for  their  relevance  to 
UMVs  and  augmenting  the  Force  by  reducing  the  operator- vehicle  ratio  and/or  optimising  interoperability. 
Six  frameworks  were  examined  more  closely,  and  it  was  shown  that  they  do  address  the  two  objectives. 
Furthermore,  the  discussion  led  to  the  discovery  of  common  and  unique  elements  of  the  frameworks. 
The  common  elements  were: 

•  Control  Theory; 

•  Hierarchy; 

•  Sensing  the  world,  own  state,  other  actors,  and  understanding  the  mission  objectives; 

•  Descriptive  analyses;  and 

•  Design  philosophy  and  guidelines. 

The  unique  elements  were: 

•  Playbook; 

•  Mathematical  analyses;  and 

•  Reduced  ratio  means  increased  uncertainty. 

3.5.1  Recommendations 

The  recommendations  focus  on  the  common  elements  while  the  unique  elements  act  as  a  reminder  to  the 
designer  as  they  consider  their  UMV  systems. 

The  recommendation  is  that  a  UMV  system  designer  should  consider  having  a  framework  guide  the  design 
process.  The  framework  should  have  elements  of  control  theory  and  hierarchy.  The  framework  should  address 
sensing  and  understanding  as  much  about  the  world,  the  actors  in  that  environment,  and  the  mission 
objectives.  The  framework  should  lead  to  descriptive  analyses,  although  mathematical  analyses  would  be 
helpful  as  well.  The  framework  should  yield  design  philosophies  and  design  guidelines  such  as  the  situation 
uncertainty  principle. 
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Appendix  3-1 :  PHASE  I  -  LITERATURE  REVIEW 

A3-1.1  LITERATURE  SEARCH  AND  PRELIMINARY  REVIEW 

The  project  Statement  of  Work  called  for  a  Phase  I  survey  to  assess  the  scope  (breath  and  depth)  of  the 
immediately  available  literature  related  to  operator-agent  interaction  for  intelligent  adaptive  interface  design. 
The  actual  searching  fields  were  beyond  suggested  areas  such  as  tele-robotics  and  human  computer 
interaction,  they  included  supervisory  control,  information  management  and  decision  support,  automatic 
manufacturing,  medical  diagnosis  and  consultation,  and  other  social  behaviour  areas,  etc. 

The  keywords  were  used  in  searching  are:  adaptive  automation,  adaptive  interface,  intelligent  interface, 
and  intelligent  user  interface.  A  number  of  databases  were  searched  and  included  in  Table  A3-1.1. 


Table  A3-1.1:  List  of  Databases  Searched 


Databases 

Starting  Year 

Number  of  Hit 

NTIS 

1964 

12 

INSPECT 

1969 

4 

Ei  Compendex 

1970 

16 

Biosis  Previews 

1969 

5 

EMBASE 

1974 

73 

Pascal 

1973 

144 

Transport  Research 

1970 

63 

Inside  Conferences 

1993 

65 

Mathscience 

1940 

239 

SciSearch 

1974 

434 

MEDLINE 

1966 

155 

Information  Science  Abs. 

1966 

202 

PsycINFO 

1887 

11 

More  than  500  abstracts  were  retrieved  from  all  databases  above,  and  there  are  127  papers  and  reports  were 
found  likely  relevant  to  the  research  domain  and  were  requested.  Totally,  82  papers  and  reports  were  reviewed 
as  they  have  different  focuses  related  to  the  topic  of  this  research,  as  included  in  Table  A3- 1.2. 
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Table  A3-1.2:  Numbers  of  Papers  in  Different  Research  Domain 


Domain 

Number 

Generic  Review  and  Discussions 

25 

Architecture/Modelling,  Technologies,  and  Implementations 

30 

Theoretical  Models  and  Empirical  Evaluations 

27 

Total 

82 

A3-1.2  PAPERS  REVIEWED  IN  DETAIL 

Of  the  literature  identified  in  the  search,  and  summarized  in  above  section,  a  number  of  papers  were  selected 
for  more  detailed  review.  This  list  was  developed  based  on  a  review  of  the  abstracts  of  the  literature  list  in 
Appendix  3-1,  which  are  directly  relevant  to  the  topic  of  the  project.  While  writing  this  report,  there  are  two 
theses  and  three  important  report  coming  in  as  ordered.  However,  due  to  the  time  constraints,  10  most  relevant 
papers  (as  list  in  Table  A3- 1.3)  were  reviewed  in  detail  and  summarized  as  below.  The  new  literature  will  be 
reviewed  against  the  current  analysis  in  this  report  in  the  next  phase  that  is  to  revise  the  draft  framework  of 
operator-agent  interaction.  The  next  section  gives  a  short  summary  of  the  paper  and  brief  comment  on  its 
applicability  to  our  themes. 


Table  A3-1.3:  Categorization  of  Literature  Reviewed  in  Detail 


Paper  Number 

© 

U 

a 

Intelligent  Interface  Introduction  and  Principles 

1,2 

>► 

QJ 

'S 

n/ 

Theoretical  Framework  an/or  Empirical  Evaluation 

3,4,  5,  6,  8,  9,  10 

hH 

a 

Operator-Interface  Interaction  Technology  and  Tools 

4,  5,  6,  7,  8,  9,  10 

S 

© 

— 

Agent  Theory  and  Technology 

1,7,8 

Adaptation,  Automation  Philosophy 

5,  6,  7,  8,  9,  10 

A3-1.2.1  Special  Issue  on  Intelligent  Interface  Technology:  Editor’s  Introduction 
(Interacting  with  Computers,  12,  pp.  315-322) 

by  Benyon,  D.R.  and  Murry,  D.M.  (2000) 

This  paper  discussed  the  reality  of  intelligent  interface  technology  (IIT):  “indirect  management”  of 
information  against  “direct  manipulation”.  Through  explaining  the  purpose  of  the  reference  model  as  a  useful 
way  of  thinking  about  IIT,  the  paper  addressed  the  reasons  why  it  is  so  difficult  to  represent  fundamental 
psychological  data  about  users.  One  of  the  reasons  for  focusing  on  psychological  models  is  that  these  are 
characteristics  which  are  most  resistant  to  change  in  people  and  which  can  vary  considerable  between 
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individuals  (as  high  as  20:1  -  i.e.,  one  person  may  take  twenty  times  as  long  as  another  to  complete  the  same 
task).  People  can  learn  domain  knowledge,  but  are  less  likely  to  be  able  to  change  fundamental  psychological 
characteristics  such  as  spatial  ability.  However,  one  of  the  difficulties  with  capturing  psychological  data  is  that 
the  only  signals  that  a  computer  can  get  are  the  sequences  of  tokens  passed  across  the  interface  and  attributes 
of  that  sequence  such  as  timing  information.  Although  this  bandwidth  will  increase,  it  still  remains  very 
narrow  compared  to  the  wealth  of  information  that  we  as  human  can  perceive. 

The  definitions  of  the  models  used  widely  in  IIT  community  were  also  given  in  the  paper.  It  is  a  good 
introduction  paper  although  there  are  more  models  in  describing  human-machine  interaction. 

•  Domain  models  are  abstract  representations  of  the  domain,  so  will  not  include  the  details. 

•  User  models  describe  what  the  system  “knows”  about  the  user.  Some  systems  concentrate  on 
developing  models  of  user  habits,  inferred  by  monitoring  user-task  interactions  over  time  (i.e.,  by 
keeping  a  dialogue  record).  Other  user  profile  data  can  often  be  most  easily  obtained  by  asking  the 
user.  Other  systems  try  to  infer  user  goals,  although  it  is  very  difficult  to  infer  what  a  user  is  trying  to 
do  from  the  data  typically  available  to  a  computer  system  (mouse  clicks  and  a  sequence  of 
commands).  The  user’s  knowledge  of  the  domain  is  represented  in  the  student  model. 

•  Interaction  model:  An  abstract  of  the  interaction  (the  dialogue  record)  along  with  mechanisms 
(such  as  a  rule-based,  a  statistical  model,  a  genetic  algorithm,  etc.)  for  making  inferences  from  the 
other  models,  for  specifying  adaptations  and,  possibly,  evaluating  the  effectiveness  of  the  system’s 
performance. 

A3-1.2.2  Steps  to  Take  before  Intelligent  User  Interfaces  Become  Real 

(Journal  of  Interacting  with  Computers,  Vol.  12,  No.  4,  pp.  409-426,  February  2000) 

by  Hook,  K.  (2000) 

This  paper  focused  on  four  challenges  for  the  intelligent  user  interface  (IUI):  usability  demands,  creating 
development  methods,  finding  useful  adaptations,  and  ensuring  maintainability.  The  concept  of  an  IUI  as  a 
means  is  to  overcome  problems  that  direct  manipulation  interface  cannot  handle:  information  overflow, 
cognitive  overload.  It  demands  better  usability  principles,  better  ways  to  improve  the  interaction,  and  better 
tools  to  survive  the  full  life  cycle  of  a  system.  The  paper  indicated  that  very  few  IUIs  that  have  succeeded 
commercially  have  done  their  very  simple  adaptations  based  on  simple  knowledge  of  the  user,  or  created  their 
adaptations  based  on  what  other  users  do  rather  than  some  kind  any  complex  inferred  model  of  the  user’s 
characteristics.  That’s  why  there  is  a  fear  that  IUI  will  violate  usability  principle  and  obscure  the  issue  of 
responsibility.  The  main  problem  is  not  whether  or  not  intelligence  at  the  interface  is  possible  or  desirable  - 
this  depends  on  a  lot  on  the  task  to  be  solved  and  the  design  of  the  total  solution  (with  both  adaptive  and  non- 
adaptive  parts).  Instead,  we  can  see  a  number  of  problems  not  yet  solved  that  prevent  us  from  creating  good 
applications.  Therefore,  there  is  a  need  to  develop: 

•  Usability  principles  for  intelligent  interfaces  (rather  than  direct-manipulation  systems)  that  do  not  lead 
users’  expectations  astray:  control,  transparency,  and  predictability;  privacy  and  trust. 

•  Reliable  and  cost-efficient  IUI  development  methods  including  functional,  data,  task  knowledge,  user, 
and  environment  analysis  first. 

•  A  better  understanding  of  how  and  when  intelligent  can  substantially  improve  the  interaction 
(i.e.,  design  practice).  Proper  evaluations  of  whether  the  system  supports  users’  real  tasks  must 
include  an  analysis  of  the  organizational  setting,  users’  activities  and  cooperation  with  each  other, 
usage  of  other  tools,  etc. 
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•  Authoring  tools  that  enable  easy  development  and  maintenance  of  the  intelligent  parts  of  the  system 
(Scalability). 

A3-1.2.3  Models  for  the  Design  of  Human  Interaction  with  Complex  Dynamic  Systems 

(Systems  Engineering  &  Management  Handbook,  1999) 

by  Mitchell,  C.M.  (1996) 

This  paper  overviewed  the  evolution  of  models  of  human-machine  systems  over  the  past  forty  years. 
From  perspectives  of  cognitive  engineering,  ecological  psychology,  and  naturalistic  decision-making, 
the  similarities  between  human-machine  systems  models  and  a  variety  of  other  recent  approaches  to 
understanding  and  aiding  human  interaction  in  real-world  systems  were  described.  The  paper  also  proposed  a 
set  of  tenets  that  characterizes  models  and  human  interfaces  whose  design  is  based  upon  them. 

The  paper  described  the  trend  of  developing  robust  models  with  the  same  levels  of  fidelity  as  system  models. 
The  vision  was  to  predict  system  and  human  behaviour  and  provide  quantitative  assessments  of  proposed 
system  design.  In  60’  and  70’,  the  crossover  control  model  and  the  optimal  control  model  (Wickens,  1984) 
focused  on  continuously  tracking  behaviour  for  fully  manual  tasks.  However,  with  Digital  Computer 
introduced,  tasks  are  supervisory  controlling  of  multiple  subsystems.  Since  then,  modeling  goal  changed  from 
pursing  the  development  of  a  global  and  analytic/computational  operator  simulation  (i.e.,  a  quantitative, 
predictive  model)  to  more  focused:  the  development  of  system  and  task  representations  that  could  be  used  for 
the  design  of  operator  interfaces  to  complex  dynamic  systems,  including  displays,  aids,  and  training  systems. 
Therefore,  the  goal  is  no  longer  to  produce  a  black-box  human  operator  simulator  that  functions  as  robustly  as 
traditional  engineering  models  of  system  hardware  and  software,  but  rather  the  development  of  a  useful 
description/prescription  of  the  system-task-operator  interactions.  Understanding  and  modeling  human 
cognition,  problem  solving,  and  decision-making  became  more  and  more  popular  and  practical  for  the  system 
design. 

The  essence  of  this  paper  is  the  emphasis  of  the  importance  of  context  as  the  central  tenet  of  Human-Machine 
Systems  Engineering.  As  Baron  (1984)  summarized  this  requirement  succinctly:  (A  human-machine  systems 
model)  ...  embodies  the  idea  that  to  model  human  performance,  one  must  model  the  system  in  which  that 
performance  is  embedded.  Human  behaviour,  either  cognitive  or  psychomotor,  is  too  diverse  to  model  unless  it 
is  sufficiently  constrained  by  the  situation  or  environment;  however,  when  these  environmental  constraints  exist, 
to  model  behaviour  adequately,  one  must  include  a  model  for  that  environment  (the  perspective  of  ecological 
interface  design).  Therefore,  the  system  design  should  be  context-oriented,  descriptive,  and  prescriptive. 

A3-1.2.4  A  Theoretical  Analysis  and  Preliminary  Investigation  of  Dynamically  Adaptive 
Interfaces 

(International  Journal  of  Aviation  Psychology,  2001,  Vol.  11,  No.  2,  pp.  169-195) 

by  Kevin  B.  Bennett,  et  al.  (2001) 

This  paper  described  a  dynamically  adaptive  interface  (DAI),  which  changes  display  or  control  characteristics 
of  a  system  (or  both)  in  real  time.  The  goal  of  this  DAI  is  to  anticipate  informational  needs  for  desires  of  the 
users  and  provide  that  information  without  the  requirement  of  an  explicit  control  input  by  the  user. 
This  research  found  that  DAIs  have  the  potential  to  improve  overall  human-machine  system  performance  if 
they  are  properly  designed.  However,  DAIs  also  have  a  very  real  potential  to  degrade  performance  if  they  not 
properly  designed.  This  study  explores  both  theoretical  and  practical  issues  in  the  design  of  DAIs.  The  relation 
of  the  DAI  concept  to  decision  aiding  and  automation  was  discussed,  and  a  theoretical  framework  for  design 
was  also  outlined. 
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In  this  paper,  a  preliminary  investigation  of  the  DAI  design  concept  was  conducted  in  the  domain  of  aviation 
(precision,  low-level  navigation).  Three  interfaces  were  evaluated  including:  non-traditional  controls  (a  force 
reflecting  stick)  and  displays  (a  configurable  flight  director)  were  developed  to  support  performance  at  the 
task;  a  standard  interfaces  (conventional  controls  and  displays),  a  candidate  interface  (alternative  controls  and 
displays);  and  an  adaptive  interface  (dynamically  between  the  standard  and  candidate  displays).  The  results 
indicated  that  significant  performance  advantages  in  the  quality  of  route  navigation  were  obtained  with  the 
candidate  and  adaptive  interfaces  relative  to  the  standard  interface;  no  significant  differences  between  the 
candidate  and  adaptive  interfaces  were  obtained;  no  significant  differences  between  the  candidate  and 
adaptive  interfaces  were  obtained.  The  implication  of  these  results  were  discussed,  with  special  emphasis  on 
their  relation  to  fundamental  challenges  that  must  be  met  for  the  DAI  concept  to  be  a  viable  design  alternative. 

This  paper  is  a  good  example  for  human-machine  interaction  from  theoretical  development  to  empirical 
investigation  to  maximize  overall  performance.  The  design  and  comparison  of  three  interfaces  is  a  typical 
method  for  such  kind  of  research.  The  results  are  also  very  interesting  as  they  raised  the  issue  of  the  dilemma 
for  automation  and  adaptation.  When,  where,  what,  why,  how  to  adapt  is  really  a  question  theoretically  and 
practically  for  better  operator-machine,  operator-agent  interaction.  The  theoretical  framework  and  research 
methodology  are  very  useful  for  other  similar  research. 

A3-1.2.5  Integrating  Perceptual  and  Cognitive  Modeling  for  Adaptive  and  Intelligent 
Human-Computer  Interaction 

(Proceedings  of  the  IEEE  Volume  90,  Issue  7,  July  2002,  pp.  1272-1289) 

by  Duric  Z.,  et  al.  (2002) 

Through  both  theoretical  analysis  and  empirical  investigations,  this  paper  is  trying  to  advocates  the  technology 
and  tools  into  interface  design  for  intelligent  human-computer  interaction  where  human  cognitive,  perceptual, 
motor,  and  affective  factors  are  modeled  and  used  to  adapt  the  human  computer  interface.  The  essence  of  the 
paper  is  to  monitor  affective  behaviour  or  emotional  behaviour  or  non-verbal  information  to  answer  W5 
(i.e.,  what  why,  where,  when,  and  how)  and  adapt  the  display  according  to  this  behaviour.  The  method  for 
interface  design  is  more  human-like  in  which  the  interface/machine  or  the  agent/automation  embedded  is 
regarded  as  another  human  assistant  who  can  monitor  perceptual  and  cognitive  levels  and  understand  the 
“partner”  or  “teammate”,  thus  react  for  better  collaboration  with  better  overall  results.  The  idea  is  to  emphasis 
the  team  collaboration,  which  is  true  and  good  in  human-human  interaction. 

“It  is  not  only  computer  technology  that  needs  to  change  to  make  such  novel  interfaces  a  reality. 
People  have  to  change  as  well  and  adapt  to  the  interface  that  the  computer  presents  them  with.  In 
the  end,  both  people  and  the  computer  have  to  understand  each  other's  intentions  and/or 
motivations,  provide  feedback  to  each  other  as  necessary,  and  eventually  adapt  to  each  other.  ” 

A3-1.2.6  Adaptive  Interfaces  for  Human-Computer  Interaction:  A  Colourful  Spectrum 
of  Present  and  Future  Options 

(IEEE  International  Conference  on  Systems,  Man  and  Cybernetics,  1995,  Volume  1,  292-297) 

byBalint,L.  (1995) 

This  paper  discussed  what  kind  of  adaptations  could  be  built  into  the  interfaces  allowing  human-computer 
interaction  from  different  aspects  of  human  behaviour:  by  its  nature,  physiological  attributes  (eye,  ears, 
fingers,  etc.),  intellectual  characteristics  (capacity,  recognition,  learning,  decision,  etc.),  knowledge  basis 
(knowing  the  environment,  the  system,  him/herself,  etc.)  and  psychological  states  (concentration,  vigilance, 
fatigue,  patience,  etc.).  As  pointed  out  in  the  paper,  that  adaptive  interfaces  should  be  capable  of: 
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•  Adjusting  the  forms  of  information  transfer; 

•  Transforming  the  information  contents; 

•  Altering/merging  modes  of  information  flow;  and 

•  Exchanging/combing  communication  media. 

The  paper  also  discussed  the  future  of  adaptive  interfaces,  the  role  of  formal  interaction  modeling, 
the  importance  of  abstract/structural  interface  hierarchy,  the  integration  of  interaction  modes  and  media, 
the  sophistication  of  interface  modularity  and  the  exploitation  of  the  advantages  in  combining/integrating 
conceptual-functional-physical  design  aspects.  It  is  a  good  reference  of  Taxonomy  of  Adaptive  Interfaces. 

A3-1.2.7  Intelligent  User  Interfaces:  An  Introduction 

(International  Conference  on  Intelligent  User  Interfaces  1999  ACM  Press) 

by  Maybury,  M.  (1999) 

This  paper  introduced  the  concept  of  intelligent  user  interfaces  (IUI)  which  aims  to  improve  the  efficiency, 
effectiveness,  and  naturalness  of  human-machine  interaction  by  representing,  reasoning,  and  acting  on  models 
of  the  user,  domain,  task,  discourse,  and  media  (e.g.,  graphic,  natural  language,  gesture).  As  indicated  in  the 
paper,  IUIs  are  multi-faceted,  in  purpose  and  nature,  and  include  capabilities  for  multi-media  input  analysis, 
multi-media  presentation  generation,  and  the  use  of  user,  discourse  and  task  models  to  personalize  and 
enhance  interaction. 

Two  important  areas  addressed  in  the  paper  are  agent  based  interaction  and  evaluation.  Research  on  agent 
technology  has  increased  in  prominence  in  applications,  which  includes  the  use  of  agents  to  express  system 
and  discourse  status  via  facial  displays,  multi-modal  communication  between  animated  computer  agents,  and 
standards  and  open  architectures  for  building  agent  based  multi-modal  interfaces  -  but  the  key  questions  are: 
what  can  and  should  an  agent  do?  How  they  should  do  it?  How,  when  and  why  should  they  interact  with 
the  user  when  doing  it?  In  terms  of  evaluation,  it  can  be  glass-box  (internal)  and  black-box  evaluation 
(end-to-end).  Criteria  for  evaluation  might  include  quantitative  ones  (e.g.,  time  to  perform  tasks,  accuracy  of 
tasks,  percent  of  inter-assessor  agreement)  as  well  as  qualitative  ones  (e.g.,  user  indication  of  utility,  ease  of 
use,  naturalness).  This  is  a  good  reference  for  the  process  of  developing  and  testing  agent  technology  based 
interfaces. 


A3-1.2.8  The  Future  of  Watchstation  Design:  Evolution  from  Single  Purpose  to  Intelligent 
Watchstations 

(2002  Command  and  Control  Research  and  Technology  Symposium,  Naval  Postgraduate 
School,  Monterey,  California,  11-13  June  2002) 

by  O’Donnell,  L.  (2002) 

The  focus  of  this  paper  is  to  address  the  issues  in  interface  design  and  the  changes  in  the  console  design  to 
support  distributed  mission  task  activities  for  joint  operations  of  global  command  and  control  systems. 
Increased  mission  demands  combined  with  smart  weapons,  automated  functions  and  increased  collaborative 
warfighter  functions  have  increased  the  multi-tasking  requirements  to  be  accomplished.  Humans  in  a 
warfighter  role  have  shifted  from  a  narrow  task  focus  within  a  narrow  job  focus  of  a  single  purpose 
watchstation  and  a  high  human-in-the  loop  interface  workload,  to  becoming  controllers  of  these  distributed 
systems  and  collaborative  activities.  In  the  paper,  current  watchstation  design  was  described  that  requires  the 
human  to  perform  manual  system  operations  in  combination  with  numerous  independent  synchronous 
activities  such  as  communications  and  adjacent  equipment  operations.  Future  watchstation  will  need  to  be 
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designed  to  support  the  work  environment  with:  increased  multi-tasking  capabilities,  dynamic  monitoring  of 
task  process,  integrated  system  designs,  and  improved  distributed  team  collaboration  task  capabilities. 
Advances  in  technology  have  enabled  the  design  of  an  effective  watchstation  design  that  will  allow  for  multi¬ 
modal  user  interfaces  best  suited  to  the  task.  Future  watchstation  designs  should  also  utilize  self-adaptive 
interfaces,  increased  visual  workspaces,  agent  technologies,  integrated  speech,  and  visual  and  direct  touch 
methods  to  reduce  the  human-interface  workload  and  streamline  the  tasks.  All  these  features  are  required  for 
future  UAV/UCAV  control  interfaces. 

The  paper  not  only  analyzed  the  current  trends  and  advantages  of  intelligent  interfaces  (watchstations), 
but  also  brought  up  a  smart  agent  taxonomy  to  construct  the  flexible,  dynamic,  scalable,  and  robust  distributed 
system  capabilities  over  system  networks  as  multiple  agent  systems.  Although  the  context  discussed  in  the 
paper  was  not  focused  on  airborne  multiple  UAV  control,  but  the  discussions  of  several  key  technologies  to 
enabling  an  intelligent  system  and  future  research  recommendations  on  intelligent  user  interface  design  can  be 
generally  applied  to  any  design  of  the  framework  for  optimal  operator  -  agent  interaction. 

A3-1.2.9  An  Architecture  for  Intelligent  Interfaces:  Outline  of  an  Approach  to  Supporting 
Operators  of  Complex  Systems 
(Human  Computer  Interactions,  3(2),  pp.  87-122) 

by  Rouse,  W.B.,  Geddes,  N.D.  and  Curry,  R.E.  (1997-1998) 

This  paper  described  a  concept  of  a  comprehensive  support  system  design  for  operators  of  complex  systems. 
A  variety  of  difficult  design  issues  as  well  as  ongoing  efforts  aimed  at  resolving  theses  issues  were  also 
addressed. 

The  main  focus  of  the  paper  is  addressing  two  methodology  issues  to  the  design:  design  methodology  and 
automation  philosophy.  Although  the  suggested  design  methodology  follows  the  traditional  human  factors 
engineering  principles,  automation  philosophy  emphasises  on  maximizing  overall  performance  by  overcoming 
human  limitations  and  enhancing  human  abilities.  The  focus  of  the  paper  (which  is  different  from  many  other 
literatures)  is  the  emphasis  of  automation  as  a  backup  -  the  default  modes  are  usually  manual  with  automation 
invoked  only  when  either  anticipated  operator  performance  is  unacceptable  or  the  operator  chooses  to 
relinquish  control.  With  the  adoption  of  this  operator-centred  automation  philosophy,  an  architecture 
including  intelligent  management  of  information  and  tasks  was  proposed.  Within  it,  the  concept  of  operator 
state  is  central  to  the  functioning  of  the  components  of  the  intelligent  interface.  The  relevant  elements  include: 
activities,  awareness,  intentions,  resources,  performance.  Another  component  is  the  interface  manager  which 
is  similar  to  an  executive’s  assistant  who  zealously  guards  the  superiors’  time  and  resources.  Although  two 
important  questions  were  raised:  when  and  how  to  automate,  there  was  still  no  answers  in  the  paper.  Another 
important  question  of  what  to  automate  was  not  addressed  in  the  paper  either.  The  paper  discussed  a  nice 
layout  of  the  architecture  for  complex  system  design,  but  it  did  not  cover  enough  cognitive  and  perceptual 
aspects  of  a  dynamic,  complex,  and  interactive  system,  especially  for  multiple  UAV/UCAV  control. 
The  concept  of  automation  control  in  the  paper  may  not  apply  to  the  supervisory  control  mode  in  the  UAV 
case,  where,  it  is  still  operator  centred  design,  but  automatic  agents/  assistants  will  play  significant  roles. 

A3-1.2.10  A  Model  for  Types  and  Levels  of  Human  Interaction  with  Automation 
(IEEE  Transactions  on  Systems,  Man  and  Cybernetics,  30  (3),  pp.  286-297) 

by  Parasuraman,  R.,  Sheridan,  T.B.  and  Wickens,  C.D.  (2000) 

A  framework/model  was  proposed  for  types  and  levels  of  human  interaction  with  automation  (Sheridan’s  10 
points  scale).  They  also  proposed  four  broad  classes  of  functions  which  automation  could  be  applied  from 
information  processing  point  of  view: 
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a)  Information  acquisition; 

b)  Information  analysis; 

c)  Decision  and  action  selection;  and 

d)  Action  implementation. 

Within  each  of  these  types,  automation  can  be  applied  across  a  continuum  of  levels  from  low  to  high, 
from  fully  manual  to  fully  automatic.  A  particular  system  can  involve  automation  of  all  four  types  at  different 
levels.  Since  automation  does  not  merely  supplant  but  changes  human  activity  and  can  impose  new 
coordination  demands  on  the  operator,  appropriate  selection  is  important  based  on  the  primary  and  secondary 
evaluative  criteria.  The  primary  criteria  look  at  human  performance  consequences:  mental  workload,  situation 
awareness,  complacency,  and  skill  degradation.  The  second  criteria  include  the  automation  reliability  and  the 
costs  of  consequences. 

Although  the  paper  considered  human  machine  interaction  mainly  from  information  process  point  of  view,  the 
proposed  model  can  be  taken  as  a  good  starting  point  for  considering  what  types  and  levels  of  automation 
should  be  implemented  in  a  particular  system.  The  paper  is  concerned  with  human  performance  in  automated 
systems  and  emphasizes  human-machine  comparison.  Automation  is  defined  as  a  device  or  system  that 
accomplishes  (partially  or  fully)  a  function  that  was  previously,  or  conceivably  could  be,  carried  out  (partially 
or  fully)  by  a  human  operator.  The  paper  also  touched  a  litter  bit  on  action  automation  -  agents  which  track 
user  interaction  with  a  computer  and  execute  certain  sub-tasks  automatically  in  a  contextually-appropriate 
manner.  However,  this  area  should  be  elaborated  more  associated  the  theory  developments  and  empirical 
investigations  (which  is  the  future  work  as  mentioned  in  the  paper).  Even  in  the  proposed  model,  the  issue  in 
whether  automation  unreliability  has  similar  negative  effects  for  all  four  stages  of  automation  needs  further 
examination.  Regarding  costs  of  decision/action  outcomes,  individual  differences  between  users  in  the  same 
interface  should  be  addressed  more,  especially  on  user  profile  building  on  the  interface  (user  modelling 
embedded  in  interface).  It  is  good  to  point  out  that  empirical  work  needs  to  be  done  to  explicitly  compare  the 
effects  on  human  performance  of  different  levels  of  automation  for  information  acquisition  and  analysis, 
in  other  words,  the  levels  of  interface  intelligence.  Overall,  this  paper  emphasised  more  function  allocation 
between  user  and  machine  regarding  automation,  but  less  was  discussed  on  operator  interface  interaction. 
How  the  automation  should  perceive,  analyze,  understand,  react,  and  collaborate  with  user  as  an  agent  or 
assistant  still  remains  untouched. 

A3-1.3  COMMENTS  ON  THE  LITERATURE  REVIEWED  IN  DETAIL 

Despite  a  fairly  significant  investment  in  research  over  the  past  decade,  there  is  still  no  generic  framework  or 
architecture  covering  all  aspects  of  human-machine  interaction  and  the  relevant  technologies.  In  particular 
from  a  Human  Factors  Engineering  perspective,  the  lack  of  empirical  investigations  on  validating  proposed 
frameworks  makes  many  designs  costly  and  ineffective.  Many  existing  models  either  focus  on  user’s  models 
or  task/domain  models,  rather  than  interaction  models. 

A3-1.3.1  What  Does  Interaction  Model  Do? 

The  supervisory  control  often  only  implicitly  implies  models  of  representing  monitoring/situation  assessment 
system  variables,  states,  or  aggregate  measures,  especially  when  supervisory  control  systems  become  more 
sophisticated.  However,  the  human  operator  models  explicitly  represent  the  domain  of  application, 
task  constraints,  and  the  flexibility  inherent  in  human  interaction  with  a  complex  system.  Thus,  the  interaction 
models  need  to  reflect  the  work  environment  and  its  dynamic  nature,  as  perceived  by  the  operator  given  the 
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current  system  state  and  current  system  goals.  The  model  of  control  activities  must  represent  at  least  three 
properties  of  both  the  control  and  controlled  systems  and  the  operator  supervising  them: 

1)  What  changes  to  the  system  the  operator  wants  to  make; 

2)  Why  the  changes  should  be  made,  with  respect  to  system  goals  and  current  state;  and  finally 

3)  How  the  needed  changes  to  the  system  can  be  made,  i.e.,  the  operator  activities  undertaken  to  bring 
about  the  desired  state. 

These  models  should  represent  the  concurrent  nature  of  the  control  activity  and  the  choices  available  to  the 
operator  given  current  system  state.  To  be  useful  to  the  design,  an  effective  model  must  be  both  descriptive 
and  prescriptive  to  describe  what  operator  actually  do  and  specifies  what  an  operator  should  do. 

Figure  A3 -1.1  illustrates  the  relations  of  user,  system,  and  the  interaction  between  them.  In  which,  user  can  act 
as  a  board  of  directors  of  a  corporation,  and  the  system  can  be  regarded  as  many  agents  assisting  the  CEO  who 
is  the  intelligent  interface.  In  order  to  help  the  board  members  to  make  decision,  through  CEO,  agents  will 
provide  necessary  information  to  keep  the  board  members  informed  what  is  currently  being  done  and  what  is 
going  to  happen  at  the  next  stage  (descriptive  and  prescriptive).  At  the  same  time,  agents  will  monitor  the 
board  members’  cognitive  workload  and  performance,  and  guard  the  resources  and  time.  Agents  can  also  learn 
from  past  experience  and  change  how  they  behave  in  given  situations  (adaptive).  Agents  will  communicate 
and  cooperate  with  other  agents  and  act  according  to  the  results  of  that  communication  (cooperative). 
Therefore,  user-system  interaction  basically  is  user-agent  interaction  for  general  cases.  Here,  agent  technology 
is  taken  as  a  means  to  facilitate  the  interaction  between  user  and  agents,  which  is  the  topic  in  next  section. 


Figure  A3-1.1:  Relation  of  User  System  Interaction. 


A3-1.3.2  What  Key  Functions  should  an  Interface  Have? 

Key  functions  within  the  user-system  architecture  include  information  management,  error  monitoring,  and 
adaptive  aiding.  One  of  the  central  knowledge  sources  underlying  this  functionality  is  an  operator  model  that 
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involves  a  combination  of  algorithmic  and  symbolic  models  for  assessing  and  predicting  an  operator’s 
activities,  awareness,  intention,  resources,  and  performance. 

The  intelligent  interface  should  also  have  many  features  to  support: 

•  More  efficient  interaction  -  enable  more  rapid  task  completion  with  less  work. 

•  More  effective  interaction  -  doing  the  right  thing  at  the  right  time,  tailoring  the  content  and  form  of 
the  interaction  to  the  context  of  the  user,  task,  dialogue. 

•  More  natural  interaction  -  supporting  spoken,  written,  and  gestural  interaction,  ideally  as  if 
interacting  with  a  human  interlocutor. 

A3-1.3.3  What  Interaction  Level  should  an  Interface  Have? 

As  pointed  out  by  Lajos  (1995),  more  complex  adaptivity  schemes  should  require  and/or  allow: 

•  More  levels  of  interaction; 

•  More  media  and  modes  (content  and  form)  of  interaction; 

•  Involvement  of  more  personal  shaping  factors  into  adaptation; 

•  Wide  choice  of  skill-based,  rule-based  and  knowledge-based  interaction  elements;  and 

•  More  modularity/hierarchical  levels  in  the  interface  structure  and  functions. 

The  moral  behind  this  is  that  people  see  not  only  with  the  eyes,  but  with  the  brain  as  well.  In  other  words, 
perception  involves  a  whole  and  purposive  cognitive  process.  From  another  perspective,  people  are  selective 
on  their  attention  [8].  In  other  words,  attention  is  not  decided  by  what  being  presented  (perception  process), 
but  what  being  decided  by  the  brain  (cognitive  process).  People  tend  to  forget  what  has  been  seen  and  what 
has  been  heard,  especially  in  a  complex,  dynamic,  and  challenging  environment.  The  proactive  feedback 
system  can  better  serve  the  purpose  of  communicate  between  user  and  interface  for  adaptation.  There  are 
certain  technologies  and  models  can  be  used  for  such  kind  of  intelligent  system. 

Other  communication  channels  between  user  and  interface  should  be  considered  as  well  such  as  verbal  and 
aural  inputs.  Multi-modal  inputs  are  good  for  monitoring  and  communicating  between  user  and  interface, 
but  the  more  information  being  provided  and  processed,  the  more  stress  and  workload  for  the  interaction. 
The  appropriate  level  and  channel  of  input  and  interaction  should  depend  on  the  task  itself.  The  reliability  and 
accuracy  for  those  user  models  and  algorithms  are  also  critical  for  the  whole  system.  It  should  also  involve 
trust  and  transparency  issues:  Trust,  when  the  agent  does  things  automatically;  Transparency,  when  the 
interface/agent  effectively  disappears,  thus,  enabling  the  user  to  interact  directly  with  the  objects  of  interest  in 
the  domain  and  to  achieve  effective  interaction  with  a  minimum  of  cognitive  effort.  A  dynamic  adaptive 
interface/agent  should  automatically  provide  information  without  the  requirement  for  control  input  by  the  user 
with  the  help  of  cognitive  inference  aid. 

Figure  A3- 1.2  illustrates  the  three  variables  of  human-machine  interfaces:  level  of  (task)  complexity,  level  of 
(interface)  intelligence,  and  level  of  (user-interface)  interaction.  Task  complexity  is  decided  by  the  nature  of 
task  (e.g.,  the  number  of  vehicles  to  control,  the  level  of  stress,  etc.).  Interface  intelligence  is  decided  by  agent 
technology,  and  it  should  cover  all  aspect  of  human  perception,  behaviour,  and  cognition.  It  is  also  related 
directly  to  automation  level  -  how  much  work  being  done  by  the  agents  automatically.  User’s  interaction 
depends  on  both  the  levels  of  task  complex  and  interface  intelligence.  Obviously,  if  the  level  of  task 
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complexity  is  high  and  automation  level  is  low,  then  the  user  would  probably  interact  more  -  but  how  much 
user  interaction  really  depends  on  the  nature  of  the  task  and  the  interface. 


Perception:  visual  verbal,  aural,  tactile,  force 

Cognition:  workload,  situational  awareness, 
complacency,  skill  degradation,  fatigue, 
frustration 

Behaviour:  manual  control  of  input  devices 

- ► 

Level  of  Complexity 

(Task  Domain) 


Figure  A3-1.2:  Taxonomy  of  Human-Machine  Interaction. 
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Appendix  3-2:  COMMENTS  ON  THEORETICAL  FRAMEWORK 
FOR  UNINHABITED  MILITARY  VEHICLES 

Erik  Hollnagel 

In  the  following  I  shall  attempt  to  comment  on  the  points  raised  in  the  letter  from  Phil  Farrell,  in  the  ongoing 
work  on  NATO  RTO  HFM  TG-017.  The  comments  refer  to  the  theoretical  framework  that  I  presented  at  the 
meeting  in  Leiden,  2003.  This  framework  is  referred  to  as  the  Extended  Control  Model  (ECOM).  The  model 
was  outlined  in  the  presentation  given  at  the  meeting  in  Leiden,  and  has  also  been  described  in  a  conference 
paper  [1]  and  a  book  chapter  in  [2].  The  pedigree  of  the  model  includes  the  description  of  the  principle  of 
contextual  control  [3],  the  initial  contextual  control  model  [4],  and  the  fully  developed  contextual  control 
model  [5]. 

These  comments  do  not  include  a  detailed  description  of  the  model.  For  details,  see  Chapter  7  (Section  7.7). 
Briefly  explained,  the  model  provides  a  framework  for  describing  how  a  joint  cognitive  system  can  maintain 
control  of  a  situation  or  a  process.  A  cognitive  system  is  defined  as  “a  system  that  can  modify  its  behaviour  on 
the  basis  of  past  experience  so  as  to  achieve  specific  antientropic  ends.”  A  joint  cognitive  system  can  be  any 
combination  of  humans  and  machines  (technological  artefacts)  or  humans  and  humans  (social  groups/ 
organisations).  The  model  invokes  the  principle  of  multiple,  simultaneous  layers  of  control.  The  layers  are 
hierarchically  organised  with  one  or  more  instances  at  each  layer.  Control  layers  differ  with  respect  to  the  time 
window  or  time  horizon  they  cover,  as  well  as  in  the  balance  between  feedback  and  feedforward  control.  The 
model  currently  describes  four  layers  of  control  called  (from  the  top  down)  targeting,  monitoring,  regulating, 
and  tracking.  It  is  applicable  both  to  single  systems  (e.g.,  a  driver  and  a  vehicle)  and  to  larger  entities  and 
organisations. 

I  should  make  clear  that  my  experience  is  primarily  from  applications  outside  the  military,  typically  process 
industries  and  transportation  (surface,  air,  space),  where  I  have  been  working  with  issues  of  automation, 
human-machine  interaction,  and  risk  and  reliability. 


A3-2.1  FORCE  MULTIPLICATION 

•  Does  the  framework/model  address  operator  to  UMV  ratio  issues? 

•  Does  the  framework/model  address  interoperability  issues? 

ECOM  will  enable  a  modelling  of  the  joint  system  (i.e.,  multiple  operators  and  multiple  UMVs),  thereby 
making  possible  an  evaluation  of  specific  allocations.  The  question  of  a  ratio  as  such  cannot  be  answered, 
since  it  will  depend  on  the  type  of  scenario,  on  situational  demands  (and  resources),  time  constraints,  urgency, 
etc.  ECOM  will  enable  a  constructive  discussion  of  allocations  for  different  types  of  scenarios,  but  not  provide 
any  automatic  answers  -  as  there  are  none! 

I  am  not  certain  what  exactly  is  meant  by  interoperability  issues.  In  computer  science  it  refers  to  the  ability  to 
exchange  and  use  information  at  different  locations  or  points  in  a  system,  such  as  a  network.  ECOM  will  not 
address  interoperability  directly,  but  will  give  guidelines  about  which  information  is  needed  -  at  different 
points  -  to  maintain  control  of  the  joint  system.  In  that  sense  interoperability  is  indirectly  addressed.  In  cases 
where  maintaining  control  is  critical,  this  may  lead  to  considerations  of  redundancy  (of  control),  hence  to 
interoperability  issues. 
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A3-2.2  UMV  SCENARIO  /  USE  CASES 

•  Is  the  theory  applicable  to  UMV  situations  (i.e.,  underwater,  sea,  land,  air,  space)? 

ECOM  is  not  domain  specific,  and  can  therefore  be  applied  to  different  types  of  UMVs.  As  noted  above, 
it  can  also  be  applied  to  different  levels  of  system  aggregation,  since  the  unit  of  analysis  is  the  joint  system. 
To  do  so  requires  a  clear  definition  of  the  system  boundaries,  since  these  are  functional  rather  than  structural 
(i.e.,  not  necessarily  defined  in  relation  to  physical  entities.) 


A3-2.3  THEORY  EVALUATION 

•  Has  the  framework/model  been  evaluated,  tested,  and  applied  to  commercial  or  military  operations? 

•  Do  you  have  an  example,  closely  related  to  UMV  situations,  where  the  theory  was  implemented? 

An  early  version  of  ECOM  has  been  used  in  a  military  context  [6]  The  model  has  also  been  used  in  research 
on  joint  driver-vehicle  systems  (automobiles)  (Hollnagel,  Nabo  &  Lau  (2003),  and  to  model  planning  and 
maintenance  at  a  nuclear  power  plant  during  a  short  outage  [7]  An  example  closely  related  to  UMV  will  be 
developed  during  the  fall,  in  collaboration  with  Robert  Taylor  and  sponsored  by  FMV,  Sweden. 


A3-2.4  REFERENCES 

[1]  Gauthereau,  V.  and  Hollnagel,  E.  (2004).  Stepping  away  from  the  centralization/decentralization 
struggle:  planning,  control,  and  adaptation  at  a  Swedish  nuclear  power  plant.  European  Management 
Journal  (in  print). 

[2]  Hollnagel,  E.  (1993a).  Models  of  cognition:  Procedural  prototypes  and  contextual  control.  Le  Travail 
humain,  56(1),  27-51. 

[3]  Hollnagel,  E.  (1993b).  Human  reliability  analysis:  Context  and  control.  London:  Academic  Press. 

[4]  Hollnagel,  E.  (2002).  Time  and  time  again.  Theoretical  issues  in  Ergonomics  Science,  3(2),  143-158. 

[5]  Hollnagel,  E.,  Nabo,  A.  and  Lau,  I.  (2003).  A  systemic  model  for  Driver-in-Control.  2nd  International 
Driving  Symposium  on  Human  Factors  in  Driver  Assessment,  Training,  and  Vehicle  Design,  July  21-24, 
Park  City,  Utah. 

[6]  Hollnagel,  E.  and  Woods,  D.D.  (2005).  Joint  cognitive  systems:  Foundations  of  cognitive  systems 
engineering.  Boca  Raton:  CRC  Press. 

[7]  Worm,  A.  (2001).  Breaking  the  barriers:  Facilitating  efficient  command  and  control  in  multi-service 
emergency  management.  8th  World  Conference  on  Emergency  Management  (TIEMS),  June  19-22, 
Oslo,  Norway. 

[8]  Bonneh,  Y.S.,  Cooperman,  A.  and  Sagi,  D.  Motion-induced  blindness  in.  normal  observers.  Nature  411, 
798-801  (2001). 
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Appendix  3-3:  REQUEST  FOR  COMMENTS  ON  THEORETICAL 
FRAMEWORKS  FOR  UNINHABITED  MILITARY  VEHICLES 

Chris  Miller,  Ph.  D. 

Smart  Information  Flow  Technologies 
2119  Oliver  Avenue  South 
Minneapolis,  MN  55405-2440 
USA 

(612)  578-SIFT  (7438) 

(612)  374-4668 

Reference:  Action  Items  from  Sweden  Meeting,  March  2004 

The  NATO  RTO  Human  Factors  and  Medicine  Panel  Task  Group  017  entitled,  “Uninhabited  Military 
Vehicles  (UMVs):  The  Human  Factors  of  Augmenting  the  Force”  is  studying  ways  to  augment  military 
Forces  by  leveraging  the  potential  advantages  of  UMVs  to  act  as  force  multipliers.  A  specific  goal  of  the  Task 
Group  is  to  produce  a  NATO  report  that  would  identify,  prioritize,  and  address  the  Human  Factors  issues 
associated  with  integrating  UMVs  into  the  Force.  The  Task  Group  (TG)  believes  that  “Augmenting  the  Force” 
will  require  research  and  development  in  the  following  areas: 

Collaborative  Work  -  Optimal  Task  Distribution 

•  Virtual  team  performance 

•  Manned/Unmanned  collaboration 

•  Interoperability 

•  Flexible  level  of  automation 

•  Optimization  of  operator/vehicle  ratio 

Control  Stations  -  Intelligent  Operator  Support 

•  Operator  functional  state  assessment 

•  Intelligent  adaptive  interfaces 

•  Cognitive  cooperation 

•  Knowledge  management  systems 

Simply  put,  the  Force  will  be  augmented  by: 

a)  Reducing  the  operator/vehicle  ratio;  and 

b)  Improving  interoperability,  and  both  require  Human  Factors  input. 

The  Task  Group  has  organized  the  Human  Factors  of  augmenting  the  Force  into  five  primary  theme  areas 
(Enclosed  is  a  short  synopsis  of  each): 

•  Theoretical  Frameworks 

•  System  of  Systems 
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•  Supervisory  Control 

•  Levels  of  Automation 

•  Advanced  Interfaces 

Theoretical  frameworks/models  were  identified  from  the  literature  as  well  as  the  Leiden  Workshop  held  in 
June,  2003  that  may  help  in  the  development  of  the  four  other  themes.  However,  we  would  like  to  solicit 
comments  directly  from  the  authors  on  their  framework’s  relevance  with  respect  to  reducing  operator/vehicle 
ratio  and  improving  interoperability. 

Therefore,  we  invite  you  to  comment  on  your  theory  with  respect  to  the  following: 


A3-3.1  FORCE  MULTIPLICATION 

•  Does  the  framework/model  address  operator  to  UMV  ratio  issues? 

•  If  so,  how  could  the  framework/model  help  reduce  the  ratio? 

PB  allows  flexible  levels  of  control  to  be  used  by  the  operator  depending  on  need,  trust,  workload,  etc. 
As  such,  in  principle,  it  permits  a  single  operator  to  do  everything  from  joystick  control  to  swarm  control  of 
any  number  of  UAVs.  Of  course,  the  level  of  aggregation  of  the  behaviour  defined  by  the  “plays”  which  the 
operator  can  plausibly  use  to  control  must  be  more  aggregated  the  more  vehicles  controlled  -  you  can 
plausibly  tell  300  UAVs  to  execute  a  “secure  area”  play,  tell  a  team  of  4  UAVs  a  specific  flight  route  to  fly  to 
maintain  surveillance  of  a  city  block,  or  provide  dynamic  joystick  inputs  (which  could  be  thought  of  as  micro¬ 
plays  -  e.g.,  “Actuate  Flaps:  30  degrees”)  for  a  single  UAV,  but  not  provide  joystick  inputs  to  300  UAVs 
simultaneously. 

•  Does  the  framework/model  address  interoperability  issues? 

•  If  so,  how  could  the  framework/model  help  improve  interoperability? 

Plays,  especially  higher  level  plays,  can  be  defined  as  abstract  functions  to  be  fulfilled  in  alternate  ways  by 
various  combinations  of  variously  capable  UAVs  -  with  the  details  associated  with  controlling  the  specific 
UAVs  buried  in  finer-grained,  lower  level  plays.  SIFT  has,  in  fact,  recently  demonstrated  the  ability  for  a  user 
to  command  a  comparatively  high  level  play  (“Overwatch”  =  sustained  surveillance  of  an  area)  without 
reference  to  the  specific  vehicle(s)  which  will  provide  it.  When  operating  at  the  higher  levels  of  the  play 
hierarchy,  the  user  simply  commands/requests  a  “service”  and  leaves  it  the  PB  to  determine  how  to  best 
provide  that  service  given  the  available  UAV  resources  -  in  our  demonstration,  potentially  multiple  instances 
of  two  types  of  heterogeneous  UAVs  (a  rotary  winged  GT-Max  and  a  fixed  wing  Dakota)  which  might  have 
different  current  locations,  fuel  resources,  onboard  sensors,  etc.  The  user  can  also  provide  constraints  that 
limit  the  range  of  lower  level  plays  that  are  acceptable  -  e.g.,  requiring  that  an  infrared  sensor  be  used  for  the 
Overwatch  and,  thereby,  eliminating  from  consideration  any  plans  that  would  involve  a  UAV  that  doesn’t 
have  that  sensor  type.  Thus,  PB  achieves  interoperability  by  building  the  knowledge  about  how  to  utilize 
different  resources  into  PB’s  Planner  and  leaving  the  user  free  to  simply  define  the  kind  of  service  s/he 
requires  and  leaving  it  to  the  PB  to  decide  how  best  to  provide  it. 


A3-3.2  UMV  SCENARIOS  /  USE-CASES 

•  Is  the  theory  applicable  to  UMV  situations  (i.e.,  underwater,  sea,  land,  air,  space)? 
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Anywhere  we  can  define  useful  aggregations/pattems  of  repeatable  behaviours  (and  codify  the  knowledge 
about  how  to  achieve  them)  into  a  hierarchical  play  representation,  PB  is  useful. 


A3-3.3  THEORY  EVALUATION 

•  Has  the  framework/model  been  evaluated,  tested,  and  applied  to  commercial  or  military  operations? 

See  Parasuraman,  [1]  for  empirical  evaluations  of  portions  of  delegation-style  interactions.  PB  has  been 
applied  in  simulation  to  multiple  UAV  tasking  scenarios  and  platforms,  a  UGV  application  and  we  are 
currently  working  on  developing  an  application  using  PB  to  allow  a  designer  to  “task”  a  human  performance 
simulation  with  play-like  mission  scenarios  in  order  to  do,  for  example,  workload  analyses  during  platform 
design. 

•  Do  you  have  an  example,  closely  related  to  the  UMV  situations,  where  the  theory  was  implemented? 

Yes,  several.  Our  current  DARPA  UAV  project,  PVACS  is  our  richest  implementation  thus  far. 

In  answering  these  questions,  you  may  wish  to  compare  and  contrast  your  theory  with  traditional  or  other 
approaches.  Also,  please  include  references  to  the  theory  if  available  (electronic  copies  preferable). 
We  anticipate  that  your  response  will  be  1  to  3  pages,  however  there  is  no  set  limit. 


A3-3.4  REFERENCES 

[1]  Miller,  C.  and  Parasuraman,  R.  (in  revision).  “Designing  for  Flexible  Human- Automation  Interaction: 
Playbooks  for  Supervisory  Control.”  Submitted  to  Human  Factors. 

[2]  Parasuraman,  R.,  Galster,  S.,  Squire,  P.,  Furukawa,  H.  and  Miller,  C.  (submitted).  A  Flexible  Delegation- 
Type  Interface  Enhances  System  Performance  in  Human  Supervision  of  Multiple  Robots:  Empirical 
Studies  with  RoboFlag.  Submitted  for  inclusion  in  IEEE  Systems,  Man  and  Cybernetics  -  Part  A, 
Special  Issue  on  Human-Robot  Interactions,  Julie  Adams  (Guest  Ed.). 

[3]  Squire,  P.,  Furukawa,  H.,  Galster,  S.,  Miller,  C.  and  Parasuraman,  R.  (Accepted)  Adaptability  and 
flexibility  are  key!  Benefits  of  the  “Playbook”  interface  for  human  supervision  of  multiple  unmanned 
vehicles.  To  be  included  in  Proceedings  of  the  2004  Meeting  of  the  Human  Factors  and  Ergonomics 
Society,  September  20-24,  New  Orleans,  LA. 

[4]  Miller,  C.,  Goldman,  R.,  Funk,  H.,  Wu,  P.  and  Pate,  B.  (2004).  A  Playbook  Approach  to  Variable 
Autonomy  Control:  Application  for  Control  of  Multiple,  Heterogeneous  Unmanned  Air  Vehicles. 
In:  Proceedings  of  FORUM  60,  the  Annual  Meeting  of  the  American  Helicopter  Society,  Baltimore, 
MD,  June  7-10. 

[5]  Miller,  C.  and  Parasuraman,  R.  (2003).  Who’s  in  Charge?;  Intermediate  Levels  of  Control  for  Robots 
We  Can  Live  With.  In:  Proceedings  of  the  2003  Meeting  of  the  IEEE  Systems,  Man  and  Cybernetics 
society,  October  5-8,  Washington,  DC. 

[6]  Parasuraman,  R.,  Galster,  S.  and  Miller,  C.  (2003).  Human  Control  of  Multiple  Robots  in  the  RoboFlag 
Simulation  Environment.  In:  Proceedings  of  the  2003  Meeting  of  the  IEEE  Systems,  Man  and 
Cybernetics  Society,  October  5-8,  Washington,  DC. 
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[7]  Miller,  C.,  Parasuraman,  R.,  Lee,  J.,  Corker,  K.,  Johnson,  L.  and  Schreckenghost,  D.  (2003).  The  Etiquette 
Perspective  for  Human-Automation  Relationships:  Applications,  Models  and  Results.  In:  Proceedings  of 
the  47th  Annual  Meeting  of  the  Human  Factors  and  Ergonomics  Society,  October  13-17,  Denver,  CO. 

[8]  Lintern,  G.  and  Miller,  C.  (2003).  Identification  of  Cognitive  Requirements  for  New  Systems. 
In:  Proceedings  of  the  47th  Annual  Meeting  of  the  Human  Factors  and  Ergonomics  Society,  October 
13-17,  Denver,  CO. 

[9]  Miller,  C.  and  Parasuraman,  R.  (2003).  Beyond  Levels  of  Automation:  An  Architecture  for  More 
Flexible  Human-Automation  Collaboration.  In:  Proceedings  of  the  47th  Annual  Meeting  of  the  Human 
Factors  and  Ergonomics  Society,  October  13-17,  Denver,  CO. 

[10]  Miller,  C.,  Goldman,  R.,  Funk,  H.  and  Parasuraman,  R.  (2003).  Delegation  Systems:  Staying  in  Charge 
of  Highly  Flexible  Automation.  In:  Proceedings  of  the  10th  International  Conference  on  Human- 
Computer  Interaction,  June  22-27,  Crete,  Greece. 

[11]  Miller,  C.,  Goldman,  R.,  Funk,  H.  and  Parasuraman,  R.  (2002).  Delegation  as  a  Model  for  Human- 
Automation  Interaction.  In:  Proceedings  of  the  3rd  International  NASA  Workshop  on  Planning  and 
Scheduling  for  Space,  October  27-29,  2002,  Houston,  Texas. 

[12]  Miller,  C.,  Pelican,  M.  and  Goldman,  R.  (1999).  High  Level  ‘Tasking  Interfaces’  for  Uninhabited 
Combat  Air  Vehicles.  In:  Proceedings  of  the  International  Conference  on  Intelligent  User  Interfaces, 
January  5-8,  Redondo  Beach,  CA. 

[13]  Miller,  C.  and  Goldman,  R.  (1997).  “Tasking  Interfaces;  Associates  that  know  who’s  the  boss.” 
In:  Proceedings  of  the  4th  USAF/RAF/GAF  Conference  on  Human/Electronic  Crewmembers,  September 
22-26,  Kreuth,  Germany. 
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Appendix  3-4:  REQUEST  FOR  COMMENTS  ON  THEORETICAL 
FRAMEWORKS  FOR  UNINHABITED  MILITARY  VEHICLES 

Patrik  Stensson 

Swedish  Air  Force 

The  theory  “The  Human  Axiom”  or 
“The  Philosophical  Framework  for  Military  Relevance” 

A3-4.1  FORCE  MULTIPLICATION 

•  Can  the  framework/model  help  in  reducing  the  operator  to  UMV  ratio? 

Yes,  but  only  indirectly  through  the  ability  of  assessing  the  requirements  for  having  appropriate  human 
control,  the  requirements  for  making  the  human  be  appropriately  in  charge.  Not  unlikely,  it  will  help  in 
increasing  the  operator  to  UMV  ratio  instead  of  reducing  it!  And,  the  main  issue  is  that  the  theory  argues 
precisely  that  the  most  valuable  military  effect  is  related  to  appropriate  human  control,  and  on  the  contrary, 
inappropriate  human  control  leads  to  a  reduced  military  effect.  That  is,  the  theory  states  that  gain  of  military 
effect,  force  multiplication,  is  achieved  by  having  appropriate  human  control.  This  is  the  concept  of  designed 
and  applied  effect. 

•  How  could  the  framework/model  help  reduce  the  operator  to  UMV  ratio? 

It  may  help  in  establishing  the  prerequisites  for  human  control  from  an  operational  perspective,  in  order  to 
make  technological  solutions  military  relevant. 

•  Can  the  framework/model  help  with  interoperability  issues? 

Yes,  since  interoperability  consists  of  a  significant  amount  of  human  interoperability,  appropriate  human 
control  is  necessary  to  achieve  interoperability. 

•  How  could  the  framework/model  help  with  interoperability  issues? 

See  above... 


A3-4.2  UMV  SCENARIOS  /  USE-CASES 

•  Does  the  theory  cut  across  UMV  situations? 

Yes,  it’s  completely  general. 

A3-4.3  THEORY  EVALUATION 

•  Is  the  framework/model  implementable  and  testable? 

I  don’t  know 

•  Do  you  have  an  example,  closely  related  to  the  UMV  situations,  where  the  theory  was  implemented? 
No!!!!!!!!!!!!!!!!!!!!! 
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Appendix  3-5:  REQUEST  FOR  COMMENTS  ON  THEORETICAL 
FRAMEWORKS  FOR  UNINHABITED  MILITARY  VEHICLES 

Phillip  Farrell,  Ph.  D. 

A3-5. 1  DESIGN  IMPLICATIONS  FOR  MULTIPLE  AGENT  SYSTEMS 

As  machines  become  more  “intelligent”,  humans  will  insist  on  interacting  with  them  as  they  do  with  other 
humans,  otherwise  humans  may  reject  the  intelligent  agent  technology  that  is  expected  to  be  part  of  future 
UMV  systems.  In  this  context  agents  may  be  human  as  well  as  software. 

A  framework  is  proposed  based  on  Perceptual  Control  Theory  [1]  to  model  multiple  non-interacting  and 
interacting  agents  in  order  to  understand  the  system  dynamics  that  would  lead  to  design  implications  for  user- 
intelligent  agent  interaction  [2].  PCT  describes  human  cognition  as  a  means-end  hierarchy  of  control  units. 
Each  control  unit  involves  the  control  of  a  perception.  At  the  lowest  levels,  control  is  subconscious  and  can 
might  be  described  by  classical  linear  control  theory.  At  the  highest  levels,  control  is  conscious,  and  deliberate 
requiring  rule-based  thinking,  logic,  and  reasoning.  Regardless  of  the  hierarchical  level,  the  control  law 
remains  the  same  -  the  output  of  each  control  loop  attempts  to  drive  the  perception  closer  to  its  internal  goal. 
The  key  advantage  of  modelling  agents  using  PCT  is  that  one  can  apply  all  the  mathematical  power  of  Control 
Theory,  which  includes  stability  and  optimization  analyses. 

It  is  assumed  that  multiple  agent  interaction  is  more  like  human-human  interaction  than  it  is  like  human- 
machine  interaction.  Table  A3-5.1  lists  some  key  differences: 


Table  A3-5.1:  Critical  Differences  between  Human-Machine  and  Multiple  Agent  Interaction 


Human  -  Machine  Interaction  (HMI) 

Multiple  Agent  Interaction  (MAI) 

Plans,  actions  and  system  states  are  known  or 
knowable  within  limits. 

Plans  and  actions  are  not  necessarily  known  a  priori,  and 
may  produce  unexpected  system  states. 

HMI  is  specific,  systematic,  and  often 
associated  with  Standard  Operating 

Procedures. 

MAI  is  fuzzy  and  there  may  be  many  means  to  achieve 
the  same  end. 

Human  has  beliefs  (assumptions)  about  the 
machine  and  the  task.  The  machine  design 
takes  into  consideration  certain  assumptions 
about  the  human  and  the  task. 

One  agent  has  beliefs  (assumptions)  about  other  agents 
and  the  task,  and  those  beliefs  may  be  hierarchical  and 
dynamic. 

Typically  trust  is  binary  -  the  machine  works 
or  the  machine  does  not  work. 

Trust  must  be  built  over  time  because  it  becomes  more  a 
function  of  mission  completion  and  success  rather  than 
based  on  an  individual  agent’s  work. 

Typically  there  are  two  levels  of  automation  - 
fully  manual  or  fully  automatic. 

Potentially  there  could  be  a  gradient  of  automation. 
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Human  -  Machine  Interaction  (HMI) 

Multiple  Agent  Interaction  (MAI) 

It  is  possible  to  know  how  the  system 
processes  information  (i.e.,  outputs  can  be 
reconstructed  from  the  inputs). 

It  may  be  difficult,  if  not  impossible  to  trace  information 
flows  in  and  amongst  multiple  agents. 

The  context  is  typically  static  or  well-defined. 

The  context  is  dynamic  and  sometimes  unknown  a  priori. 

Only  the  human  is  truly  autonomous. 

Potentially  there  could  be  multiple  truly  autonomous 
agents. 

From  a  control  theory  perspective,  human-machine  interaction  may  be  analysed  by  treating  the  machine  as  a 
simple  input-output  transfer  function  with  some  known  disturbances  -  this  problem  is  relatively  easy  to  solve. 
On  the  other  hand,  multiple  agent  interaction  must  be  treated  as  independent  (truly  autonomous)  control  loops 
that  interact  through  a  common  portion  of  the  world  (often  called  the  interface).  Conceptually,  this  model 
would  produce  a  different  set  of  dynamics  that  are  unstable  and  not  easily  predictable.  The  clear  advantage  is 
that  multiple  agents  have  the  potential  to  multiply  the  force. 

Two  interacting  agents  as  well  as  a  generic  multiple  agent  model  was  analysed  using  mathematical  control 
theory  techniques  and  the  following  are  a  list  of  design  recommendations  that  come  from  this  modelling 
exercise: 

•  Closed-loop  feedback  modelling  techniques  can  be  used  in  the  design  of  multiple  agent  systems. 

•  Designers  should  consider  goals,  sensing  and  decision-making  strategies,  and  world  states  as  part  of 
their  system  design. 

•  Agents  should  act  on  separate  states,  while  gathering  data  from  all  sources. 

These  design  principles  have  been  applied  to  UMV  research  studies  on  the  design  of  intelligent  adaptive 
interfaces,  and  selecting  crews  for  UAV  operations.  The  third  design  philosophy  was  applied  to  information 
management  business  rules  for  the  Multi-National  Experiment  4  on  Effects  Based  Operations  with  US  Joint 
Forces  Command  as  the  experiment  lead.  This  experiment  involved  over  one  hundred  players  over  a 
distributed  network  from  countries  around  the  world.  The  interface  design  and  business  rules  were  critical  in 
order  to  successfully  collaborate  and  conduct  the  experimental  operation. 

This  paper  models  intelligent  agents  using  Perceptual  Control  Theory.  Using  standard  mathematical  control 
theory  techniques,  conditions  for  stability  are  derived  for  two-agent  and  multiple  agent  interactions. 


A3-5.2  FORCE  MULTIPLICATION 

•  Does  the  framework/model  address  operator  to  UMV  ratio  issues? 

Yes.  The  theory  has  the  potential  to  determine,  a  priori,  conditions  for  stability  for  any  operator/vehicle 
configuration. 
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•  If  so,  how  could  the  framework/model  help  reduce  the  ratio? 

The  theory  can  show  that  as  one  increases  the  number  of  intelligent  agents,  there  will  be  as  many  regions  of 
local  instability  as  there  is  stability.  Thus,  the  design  would  need  to  be  constrained  more  and  more  in  order  to 
maintain  a  stable  trajectory  through  the  state  space.  The  theory  has  the  potential  to  show  optimal  trajectories 
through  the  state  space  as  the  ratio  is  reduced. 

•  Does  the  framework/model  address  interoperability  issues? 

Yes. 

•  If  so,  how  could  the  framework/model  help  improve  interoperability? 

One  of  the  design  implications  of  this  stability  analysis  is  that  multiple  agents  can  gather  information  from 
each  other  and  the  world,  but  must  act  on  separate  parts  of  the  world.  This  principle  is  best  illustrated  with  a 
wheel  barrel  example.  Two  people  pushing  one  wheel  barrel  may  have  similar  goals  and  perceptions,  but  if 
their  actions  are  not  precisely  coordinated  then  the  system  will  quickly  become  unstable.  Similarly,  two  forces 
may  want  the  information  from  sensors  onboard  a  UMV,  but  if  they  attempt  to  control  the  vehicle  and/or 
sensors  simultaneously  (or  even  time  diviplexed)  there  is  great  potential  for  conflicts.  What  this  means  for 
interoperability  is  that  standards  might  be  required  to  ensure  proper  sensing  of  information,  and  action  must  be 
coordinated  (perhaps  proceduralized)  to  ensure  de-confliction. 


A3-5.3  UMV  SCENARIOS  /  USE-CASES 

•  Is  the  theory  applicable  to  UMV  situations  (i.e.,  underwater,  sea,  land,  air,  space)? 

Yes.  The  paper  describes  generic  multiple  agent  interactions.  It  does  not  distinguish  between  animate  and 
inanimate,  human  or  machine.  In  order  to  do  the  mathematics,  however,  it  does  make  a  fundamental 
assumption  of  linearity  -  and  we  know  that  these  UMV  situations  are  not  linear.  Control  theorists  have 
methods  to  deal  with  non-linear  systems  including  piece-wise  linearization.  Thus  the  linear  stability 
conditions  only  approach  the  real  stability  conditions  as  the  linear  pieces  reduce  in  size.  Lyapunov  approaches 
to  deriving  stability  conditions  provide  a  global  solution  to  nonlinear  dynamical  systems,  however,  the 
challenge  is  to  derive  (often  stumble  onto)  a  Lyapunov  function  that  satisfy  certain  energy  criteria.  I  think  the 
best  the  Theory  can  do  is  provide  heuristics,  but  it  can  provide  it  to  any  UMV  situation. 


A3-5.4  THEORY  EVALUATION 

•  Has  the  framework/model  been  evaluated,  tested,  and  applied  to  commercial  or  military  operations? 

Yes.  This  first  application  of  these  design  considerations  hope  to  be  in  the  Adaptive  Intelligent  Agent  project 
conducted  by  DRDC  Toronto.  The  theory  will  also  be  applied  to  a  UAV  crew  selection  methodology. 
Also,  the  theory  has  been  applied  to  the  business  rules  for  Multi-National  Experiment  4.  That  is,  all  can  view 
information,  but  only  specific  people  have  write  permissions  to  the  database.  A  multiple  UAV  interface  has 
been  designed  [3]  based  on  this  framework,  and  will  be  experimented  on  in  2006. 

•  Do  you  have  an  example,  closely  related  to  the  UMV  situations,  where  the  theory  was  implemented? 

See  above. 
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Appendix  3-6:  REQUEST  FOR  COMMENTS  ON  THEORETICAL 
FRAMEWORKS  FOR  UNINHABITED  MILITARY  VEHICLES 

Dr.  Axel  Schulte 

A3-6. 1  FORCE  MULTIPLICATION 

•  Does  the  framework/model  address  operator  to  UMV  ratio  issues? 

The  framework  advocated  by  the  Munich  University  of  the  German  Armed  Forces,  represented  by 
Prof.  Dr.  Axel  Schulte  shall  be  denoted  as  Cognitive  Automation  (CA).  CA  as  itself  does  not  address  operator 
to  UMV  ratio  issues  explicitly  since  it  is  a  general  approach  towards  improving  system  performance,  to  begin 
with.  The  basic  idea  behind  CA  is  to  mimic  human  performance  including  knowledge-based  behaviour  on  the 
machine  side  by  a  knowledge-rich  system  incorporating  explicit  goals  for  acting,  a  comprehensive  situation 
understanding  and  planning/problem-solving,  decision-making  and  acting  on  behalf  of  the  machine.  In  turn, 
such  a  cognitive  system  being  built  following  an  according  specification,  will  certainly  affect  the  operator  to 
UMV  ratio.  There  are  several  ways  to  introduce  CA  in  to  a  work  system  (i.e.,  operator(s)  controlling 
UMV(s)).  This  can  be  done  either  as  cognitive  assistant  system,  supporting  the  operator’s  work  tasks  by 
taking  workload  off  the  operator,  or  as  an  autonomously  acting  unit  taking  over  certain  comprehensive  tasks 
and  teaming  up  with  the  human.  So,  properly  designed  CA  will  supplement  human  teams  and,  thereby,  bear 
the  potential  to  affect  operator  to  UMV  ratio. 

•  If  so,  how  could  the  framework/model  help  reduce  the  ratio? 

If  this  is  a  goal  of  automation,  which  certainly  is  true  in  many  cases,  the  answer  is  yes. 

•  Does  the  framework/model  address  interoperability  issues? 

One  characteristic  of  CA  as  defined  above  is  the  explicit  representation  and  processing  of  domain  relevant 
knowledge  including  models  for  situation  understanding  and  goals  for  acting.  With  respect  to  the 
interoperability  issue,  there  might  be  distinguished  between  several  levels  of  interoperability,  i.e.,  on  the  low 
level  of  common  protocols  (e.g.,  Link  16)  as  well  as  on  the  high  level  of  a  common  understanding  of 
situations,  of  tasks,  and  of  rules  of  engagement  within  situational  contexts.  CA,  enriched  by  respective  domain 
knowledge,  will  in  principle  be  able  to  handle  problems  covering  the  full  scope  of  these  levels  and,  in  turn, 
will  be  enabled  to  take  over  tasks  from  humans  in  a  supportive  or  even  replacing  manner.  Thereby,  the  effect 
of  CA  upon  interoperability  issues  can  be  a  tremendous  one.  Hence,  the  problem  of  knowledge  elicitation  and 
knowledge  representation  is  still  an  active  research  issue. 

•  If  so,  how  could  the  framework/model  help  improve  interoperability? 

To  answer  this  question,  it  should  be  made  clear,  that  CA  is  a  systems  engineering  approach  to  intelligent 
systems  design.  The  effectiveness  of  such  a  system  is  solely  determined  by  the  right  ontology  represented  by 
the  knowledge  put  in  during  the  design  process.  Properly  designed,  we  can  expect  a  positive  effect  upon 
interoperability. 


A3-6.2  UMV  SCENARIOS  /  USE-CASES 

•  Is  the  theory  applicable  to  UMV  situations  (i.e.,  underwater,  sea,  land,  air,  space)? 
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Simple  answer,  yes,  since  CA  is  a  general  framework.  Of  course,  each  of  the  mentioned  domains  has  got  it’s 
own  peculiarities.  But,  they  all  have  in  common,  that  there  will  be  a  complex  dynamic  situation,  tactical  or 
whatsoever,  needed  to  be  interpreted  and  understood,  that  there  are  goals  for  acting  to  be  considered,  and  that 
there  is  some  demand  upon  problem-solving  and  decision-making.  Again,  it  is  a  matter  of  the  kind  of 
knowledge  you  put  into  the  system,  whatsoever  functionality  will  be  provided.  Currently  the  concept  has  been 
proven  in  the  aeronautical  domain. 

A3-6.3  THEORY  EVALUATION 

•  Has  the  framework/model  been  evaluated,  tested,  and  applied  to  commercial  or  military  operations? 

Yes,  here  is  a  very  brief  summary  of  activities: 

•  CASSY  -  Cockpit  Assistant  System:  World’s  first  comprehensive  knowledge-based  pilot  assistant 
system  for  civil  air  transportation.  Functional  prototype  successfully  flight  tested  in  1994  with 
operational  airline  captains. 

•  CAM  A  -  Crew  Assistant  Military  Aircraft.  Follow  up  activity  of  CASSY  designed  for  military 
tactical  air  transport  missions,  including  tactical  situation  assessment  and  low-level  flight  guidance 
functions.  Simulator  tested  in  1998.  Flight  tested  in  2000  with  operational  German  Air  Force  pilots. 
Some  minor  product  spin-off  for  the  A  400  M  program. 

•  TIMMS  -  Tactical  Information  and  Mission  Management  System.  Carries  many  of  the  CAM  A  ideas 
into  the  Air-to-Ground  attack  domain,  covering  operator  support  in  an  networked  air  warfare 
scenario,  addressing  some  interoperability  issues.  TIMMS  was  simulator  tested  in  2000  as  a 
functional  prototype  by  air  force  pilots.  Later  on  it  was  adapted  to  a  Eurofighter  cockpit  avionics 
environment  and  some  certification  issues  were  addressed. 

•  PILAS  -  Assistant  system  for  helicopter  emergency  medical  service  mission.  CA  serves  a  general 
approach  to  the  system  design. 

•  MiRA  -  Forthcoming  project  on  Military  Rotorcraft  Assistant. 

•  Do  you  have  an  example,  closely  related  to  the  UMV  situations,  where  the  theory  was  implemented? 

Yes,  currently  we  are  working  on  some  UAV  activities,  in  brief: 

•  COSY  flight  -  Cognitive  System  for  the  flight  domain.  Autonomous  guidance  system  for  a  single¬ 
ship  URAV  mission.  Project  was  set-up  in  order  to  prove  concept  of  the  implementation  framework 
COSA  (Cognitive  System  Architecture).  Finished  by  now. 

•  Currently  COSA  is  being  used  for  the  implementation  of  an  autonomous  guidance  system  for  a  multi¬ 
ship  SEAD  UCAV  mission.  Main  focus  is  the  implementation  of  machine-machine  co-operation 
capabilities  according  to  the  approach  of  CA.  Will  be  tested  in  simulation. 

•  Development  of  a  mini-UAV  field  demonstration  of  cognitive  and  co-operative  capabilities  on  the 
machine  side  as  well  as  intelligent  operator  assistance  in  a  ground  control  station.  The  vehicles  are 
one  model-scale  rotorcraft  UAV  and  one  fixed-wing. 

•  Manned-unmanned  teaming  -  forthcoming  project  on  integration  of  UAVs  into  military  helicopter 
missions  in  a  network  centric  operations  context. 

References  to  particular  publications  can  be  provided  on  demand. 

Prof.  Dr.  Axel  Schulte.  Bath,  UK.  May  2005. 
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Appendix  3-7:  REQUEST  FOR  COMMENTS  ON  THEORETICAL 
FRAMEWORKS  FOR  UNINHABITED  MILITARY  VEHICLES 

Iain  S.  MacLeod 

Defence  College  of  Management  and  Technology 
UK  Defence  Academy 
Shrivenham 

System  Process/Task  Organisational  Model  for  HF  V&V 

A3-7.1  INTRODUCTION 

The  author  believes  that  if  the  Task  Group  017  remit  is  truly  to  be  “The  Human  Factors  of  Augmenting  the 
Force”  we  should  produce  outcomes  that  have  enough  practicality  to  allow  their  consideration  applicability  in 
augmenting  force  effectiveness  in  the  ‘real’  world.  For  that  reason  I  believe  that  the  following  should  apply: 

•  Theoretical  Frameworks  should  be  formed  from  Theoretical  Models  and  should  have  enough 
substance  that  they  can  be  tested  in  reality  [believe  that  Leiden  workshop  had  a  strong  emphasis  on 
Models]. 

•  Following  from  the  above,  any  Theoretical  Model  should  be  based  on  generic  truth(s)  underlying 
human  systems  performance  in  the  real  world  [both  inhabited  and  uninhabited]  in  that  it  can  be  seen 
to  encompass  any  associated  theoretical  framework.  Whilst,  the  use  of  UMVs  within  a  network 
enabled  community  will  still  involve  the  issues  of  interoperability  that  are  currently  recognised  [1,2] 
(for  example,  shared  situational  awareness,  trust,  the  syntactic,  semantic,  and  pragmatic),  for  UMVs 
there  will  be  some  important  nuances  and  differences  to  the  issues  as  associated  with  inhabited 
systems. 

•  It  is  likely  that  if  a  Theoretical  Models  are  True’  there  will  be  few,  maybe  only  one! 

•  A  Theoretical  Model  architecture  must  address  its  capability  to  support  for  a  Theoretical  Framework 
architecture  or  architectures. 

•  Theoretical  Frameworks  are  only  theoretical  until  tested  when  they  then  become  a  manifestation/ 
representation  of  some  multi-dimensional  aspect  of  reality. 

•  If  a  framework  is  to  be  related  to  reality  it  should  consider  selective  combinations  of  Process, 
Context,  Situation,  Mission,  Goal(s)  [possibly  both  Strategic  and  Tactical],  Force  Structure, 
Organisations,  Cultural  influences,  the  role(s)  of  technology,  required  system  functions  and 
performance,  considered  level  of  abstraction,  temporal  issues,  roles  of  personnel,  jobs,  teamwork, 
tasks,  and  individual  actor  /  entity  properties  and  activities.  One  example  considering  generic  mission 
process  associated  target  recognition  tasks  is  introduced  in  Reference  3. 

•  For  the  considered  application  of  any  framework  the  prime  underlying  tenet(s)  to  be  examined  should 
be  made  specific,  i.e.,  Command,  control,  workload,  co-ordination,  teamwork  quality  to  name  but  a  few. 

•  It  is  important  that  the  consideration  [possibly  a  verification  and  validation  process]  of  a  framework  is 
made  both  statically  and  dynamically  -  statically  to  ensure  that  it  is  basically  logical  and  fit  for 
intended  purpose,  dynamically  to  examine  its  fit  to  reality.  Here  the  role  of  forms  of  Measures  of 
Effectiveness  and  Performance  are  important. 
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A  required  quality  of  human  work  is  always  associated  with  an  understanding  and  acceptance  of  work  goals, 
team  and/or  individual  capabilities,  appropriate  skills,  and  a  relevant  knowledge  plus  the  means  to  apply  that 
knowledge  within  the  time  constraints  and  environments  influencing  the  work. 


A3-7.2  THE  SYSTEM  PROCESS  /  TASK  ORGANISATIONAL  MODEL  FOR 
HF  V&V 

The  System  Process/Task  Organisational  Model  for  HF  V&V  presented  at  Leiden  was  shown  using  an 
embedded  Theoretical  Framework  of  Task  Organisation.  In  the  author’s  view  that  framework  always  implies 
levels  of  consideration  on  selected  combination  as  discussed  previously. 

One  of  the  problems  in  the  understanding  of  human  work  is  the  meaningful  association  of  theoretical 
interpretations  [or  hypothetical  constructs]  with  a  needed  understanding  for  the  application  of  human  work 
practices.  It  is  argued  that  the  association  of  generic  process  with  active  practises  is  one  avenue  towards  that 
understanding.  Furthermore,  derived  system  functions,  constraints,  and  architecture  support  the  quality  of 
satisfaction  of  the  process(es)  existing  to  allow  system  capability  and  not  necessarily  a  system  goal. 
In  contrast,  consideration  at  the  task  level  (whether  that  is  at  a  higher  level  consideration  of  force/system  tasks 
or  at  the  level  of  individual  human  tasks)  always  implies  a  goal  driven  application  of  appropriated  effort  to  a 
selection  of  available  functions.  Thus,  if  a  system  is  considered  dynamically  it  is  with  a  task  related 
perspective;  if  statically  it  is  more  at  a  function/constraint/architecture/required  capability  system  perspective 
of  consideration  or  the  consideration  of  an  underlying  generic  process.  For  a  high  level  discussion  on  this  area 
see  Reference  3. 

A3-7.3  QUESTION  /  RESPONSE 

Considering  the  issues  that  are  to  be  addressed  with  relation  to  the  proposed  theory  or  Theoretical  Model, 
Table  A3-7.1  gives  some  high  level  address. 
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Table  A3-7.1:  Response  of  ‘System  Process  /  Task  Organisational  Model  for  HF  V&V’  to  Questions 


Question 

Response  on  MODEL 

Force  Multiplication 

Does  the  framework/model  address  operator  to  UMV 
ratio  issues? 

Yes. 

By  allowing  consideration  as  required  through 

If  so,  how  could  the  framework/model  help  reduce  the 
ratio? 

frameworks  considering/evaluating  issues  such  as 
Automation,  Autonomy,  Manning,  and  Personnel 
within  the  force  to  examine  questions  of  force 
effectiveness  against  such  as  operator/UMV  ratios. 

Does  the  framework/model  address  interoperability 
issues? 

Frameworks  would  need  to  be  evolved  to  address 
specifics  of  interoperability  and  other  issues. 

If  so,  how  could  the  framework/model  help  improve 
interoperability? 

However,  the  model  does  allow  for  that  in  that  the 
‘Task  Organisational  Model’  example  of  a 
Framework  within  the  overall  model  can  be 
developed  to  consider  the  particular  tenets  to  be 
examined.  Interoperability  is  a  many  layered  issue 
depending  on  force  structure,  the  organisational 
level  of  the  force  being  considered,  the 
standardisation  of  procedures,  and  communication 
protocols  [human  and  machine]  to  name  but  a  few 
issues  [1,2]. 

UMV  Scenarios/Use-Cases 

Is  the  theory  applicable  to  UMV  situations  (i.e., 
underwater,  sea,  land,  air,  space)? 

The  model  is  generic  enough  to  encompass  any 
framework  considered  scenario  or  its  address 
through  particular  Use  Cases. 

Theory  Evaluation 

Has  the  framework/model  been  evaluated,  tested,  and 
applied  to  commercial  or  military  operations? 

Do  you  have  an  example,  closely  related  to  the  UMV 
situations,  where  the  theory  was  implemented? 

Yes  -  against  many,  but  applications  mainly 
implicit  rather  than  explicit. 

No  -  examples  only  with  relation  to  manned 
systems,  but  within  coalition  operations  it  is 
possible  to  consider  individual  members  of  the 
coalition  as  semi  autonomous  and  therefore  to  have 
some  properties  similar  to  UMVs. 

Some  forms  of  Framework  Models  (as  apposed  to  Local  Models!)  already  suggested  can  be  substituted/ 
incorporated  into  the  Theoretical  Framework  level  onto  the  Process  of  the  ‘System  Process/Task 
Organisational  Model  for  HF  V&V’  Theoretical  Model  -  examples  are: 

•  (Perceptual)  Control  Theory  approaches; 

•  Delegation  using  Task  Hierarchy; 

•  Responsibility  -  Authority  -  Competency; 


3-48 


RTO-TR-HFM-078 


dMNATO 
WP  I  OTAN 


THEORETICAL  FRAMEWORKS 


•  Layers  of  Control; 

•  Logical/Intuitive  channel  model  for  inducing  trust; 

•  Systems  Engineering  Approaches; 

•  Theories  of  agency; 

•  Mission  Data  System  (software  framework,  goals,  states  and  constraints);  and 

•  Rule-based  automation  (recognition-primed  decision-making). 

It  is  the  generating  of  agreed  rules  for  applying  particular  Framework  Models  to  the  generic  Process/Task 
Organisational  Model  that  have  to  be  formulated. 


A3-7.4  AUGMENTING  REALITY 

Back  again  to  real  world  reality  -  resulting  from  the  form  of  Theoretical  Models  and  Frameworks  that  are 
adopted  we  should  decide  on  the  form/type  of  scenarios,  functions  (plus  associated  performance),  constraints, 
and  capabilities  of  our  UMV  from  the  HF  perspective.  We  now  have  to  bring  these  into  the  reality  of  an 
engineering  life  cycle  for  the  UMV  system. 

Here  we  should  be  examining  the  System  Engineering  (SE)  Models  and  their  suitability  to  adopt  the  UMV  HF 
properties  we  have  arrived  at.  All  SE  models  are  poor  at  considering  HF  at  the  early  phase(s)  of  the  Life  Cycle 
(i.e.,  variously  termed  Concept/  user  requirements  definition/advanced  studies  and  preliminary  analysis,  etc.). 
If  HF  is  to  become  a  mainstream  discipline  in  its  effects  on  the  quality  of  systems  design  it  must  become  part 
of  the  logical  specification  of  system  functional  requirements  (currently  HF  mainly  dwells  on  system 
Constraints)  and  have  influence  on  the  system  architecture.  At  the  moment  HF  mainly  dwells  on  attending 
retrospectively  to  the  PRODUCTS  of  others  work.  Thus  it  is  almost  impossible  to  make  an  HF  CASE  (like  a 
Safety  Case  or  R&M  Case)  as  there  are  no  requirements  to  which  HF  can  trace  and  show  its  progressive 
address  up  to  system  acceptance  and  beyond. 

Some  SE  Standards  worth  a  look  (in  the  order  of  their  greatest  attention  to  HF)  are: 

•  INCOSE  Systems  Engineering  Handbook,  V2,  July  2000; 

•  IEEE  1220-1988,  Standard  for  Application  and  Management  of  the  Systems  Engineering  process, 
Full  Standard  version  1.0,  July  27  1998;  and 

•  ISO  15288,  WD4,  Life  Cycle  management  -  System  life  Cycle  Processes,  16  February  1999. 
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Chapter  4  -  SYSTEM  OF  SYSTEMS 

Chapter  Lead:  G.  Boucek 

Contributors:  G.  Boucek,  A.  de  Reus,  P.  Farrell,  A.  Goossens,  D.  Graeber, 

B.  Kovacs,  A.  Langhorne,  K.  Reischel,  C.  Richardson,  J.  Roessingh, 

G.  Smith,  P.  Svenmarck,  A.  Tvaryanas,  M.  van  Sijll 

4.1  INTRODUCTION 

Once  dismissed  as  novel  technology  that  would  never  be  useful  within  a  dynamic  environment,  uninhabited 
military  vehicles  (UMVs)  are  being  developed  in  greater  numbers  and  growing  sophistication  as  the  modem 
military  strives  for  greater  persistence  over  the  battlefield,  more  real-time  intelligence,  and  the  ability  to  strike 
heavily  defended  targets.  New  system  architectures  designed  for  interoperability  are  being  developed  to 
integrate  multiple  platforms  into  a  common  mission  control  element  giving  the  war-fighter  access  to  a  large 
volume  of  real-time  information.  Couple  that  with  a  geographically  dispersed  command  and  control  stmcture 
and  the  UMV  operator  is  faced  with  unique  challenges  as  traditional  inner-loop  control  tasks  are  replaced  with 
battle  management  and  command  coordination  consistent  with  an  effects  based,  network  centric  environment. 
The  end  result  is  an  entire  set  of  new  Human  Factors  related  challenges  facing  developers  to  ensure  successful 
human  systems  integration.  Resolving  issues  associated  with  connectivity,  knowledge  and  action  consistency, 
and  transfer  of  control  have  taken  center  stage  along  with  traditional  Human  Factors  issues  related  to 
information  management,  information  processing,  decision  aiding,  levels  of  autonomy,  command  and  control, 
manpower  and  skills,  and  training.  To  be  successful,  these  issues  must  be  addressed  during  the  early  stages  of 
systems  engineering  to  ensure  proper  human-centered  development  of  UMV  systems  within  a  system-of- 
sy stems  architecture. 

It  begins  with  understanding  the  concept-of-operation  in  which  UMVs  will  operate  and  then  identify  mission 
system  requirements.  As  Bruce  Clough  [27,  Chapter  7]  correctly  states,  “The  hardest  part  of  making  a 
decision  isn’t  deciding,  it’s  knowing  what  to  decide  with.”  What  is  the  situation  and  how  best  can  decision 
aiding  be  applied?  Clough  continues  with  another  lesson  learned,  “Best  autonomy  method  used  is  related  to 
task  to  accomplish,  there  is  no  optimal  method  for  any  task.”  All  too  often  Human  Factors  engineers  and 
designers  try  to  make  the  leap  from  theory  and  principles  directly  to  developing  user  interface  concepts 
without  a  clear  understanding  of  the  operator  tasks  and  requirements.  The  end  result  tends  to  be  concepts  that 
are  not  supported  by  the  system  architecture,  or  are  not  conducive  to  the  real  world.  Following  a  disciplined 
Systems  Engineering  approach  that  combines  top-down  requirements  development  with  a  bottoms-up  rapid 
prototyping  capability  should  result  in  a  human-centered  design  that  is  both  optimal  and  credible.  Using  rapid 
prototyping  tools  that  provide  standard  widgets,  display  templates,  and  auto-code  generation  allows  the  user 
interface  designer  to  produce  concepts  that  can  be  evaluated  early  and  often  by  the  operator  as  well  as 
integrated  directly  with  the  final  mission  control  system. 

The  following  sections  within  this  chapter  address  Human  Factors  issues  and  future  research  associated  with 
human  information  processing  and  information  management  within  the  context  of  command  and  control  and 
network  centric  operations.  In  addition,  issues  associated  with  migration  of  operator  control  and  the  impact  on 
teams,  interoperability,  and  situation  awareness  are  discussed.  The  chapter  concludes  with  a  section  on 
manpower  and  skills  which  addresses  operator  selection  and  training. 
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4.2  COMMAND  AND  CONTROL:  HUMAN  INFORMATION  PROCESSING  AND 
TRUST  IN  TIME  DELAYED  SYSTEMS 

4.2.1  Human  Information  Processing  (HIP)  Capabilities,  Limitations,  and  Detractors 

4.2.1. 1  Human  Information  Processing  Models 

Network  centric  capabilities  interweaving  sensors,  humans,  and  decision  aids  will  yield  increases  in  sources 
and  quantity  of  information.  These  advances  will  place  heightened  cognitive  demands  on  UMV  operators 
performing  C2  tasks.  The  increased  cognitive  demands  associated  with  a  dependence  on  large  complex  data 
sets  has  been  hypothesized  to  result  in  information  saturation  and  higher  levels  of  perceived  mental  workload 
by  the  National  Science  Foundation  (NSF)  and  the  National  Research  Council  (NRC)  [1].  The  challenge  is  no 
longer  the  lack  of  information,  but  instead  finding  the  required  information  at  the  right  time.  With  the  increase 
in  available  information,  UMV  operators  performing  C2  tasks  will  be  stretched  to  accommodate  the 
computational  demands  of  complex  network  centric  technology.  To  better  understand  the  potential  for 
overtaxing  the  UMV  operator’s  cognitive  capabilities  a  review  of  Human  Information  Processing  (HIP)  theory 
and  associated  supervisory  control  issues  are  presented  below. 

Human  information  processing  models  have  been  used  as  a  framework  to  describe  cognitive  processes  and 
characterize  interactions  with  the  environment.  These  models  provide  insight  into  cognitive  process  that  occur 
when  an  individual  perceives  information  from  the  environment,  acts  on  that  information,  and  responds  to  the 
environment  [2].  The  HIP  models  depict  a  sequence  of  serial  processing  stages  where  information  from 
previous  stages  is  manipulated,  transformed,  and/or  combined  with  other  information  before  passing  to  the 
next  stage. 

The  HIP  flow  begins  in  the  sensory  stage  when  a  sense  organ  encounters  a  stimulus  that  is  within  its 
capabilities  and  of  sufficient  intensity  to  initiate  processing.  At  this  stage,  if  attention  is  diverted  from  the 
stimulus,  it  is  stored  in  a  short-term  memory  store  (STSS).  Sensory  storage  is  available,  but  is  temporary  and 
limited  in  terms  of  capacity  and  decay  [3].  When  a  stimulus  is  attended  to,  it  enters  the  perception  stage  where 
meaning  is  attached  to  its  attributes  to  aid  in  detection,  identification,  and  recognition  of  the  stimulus 
(e.g.,  sound  is  detected,  identified  as  warning,  and  recognized  as  critical).  Perceptual  processing  occurs 
quickly  and  automatically  with  minimal  attentional  resources  driven  by  sensory  based  bottom-up  processing 
and  top-down  processing  via  long-term  memory  (LTM)  [3].  Depending  on  the  stimulus’  attributes,  either 
bottom-up  or  top-down  processing  may  dominate.  For  example,  when  a  stimulus’  characteristics  are 
ambiguous,  learned  rules  and  skills  supplement  the  information  required  to  identify  and  recognize  information 
(e.g.,  radar  image  identified  as  a  critical  target).  After  perception,  processed  information  then  enters  the 
decision  making  and  execution  stage  [4].  Depending  on  the  decision  maker,  task  and  environment,  individuals 
may  rely  on  different  decision  making  processes  [5].  An  individual  may  utilize  one  or  more  means  for 
decision  making  including  1)  automatic,  perceptual,  pattern  matching  processes,  2)  controlled,  analytic 
decision  making  processes,  or  3)  heuristics  and  biases.  Each  of  these  requires  the  use  of  memory  processing  to 
gather  information  from  LTM  and  actively  rehearse  it  in  working  memory  (WM).  Once  a  decision  has  been 
reached,  an  action  is  selected  and  a  motor  response  is  executed.  From  the  motor  response,  new  stimuli  emerge 
and  the  process  begins  over  again  creating  a  continuous  closed  feedback  loop  [4]. 

Human  information  processing  stages  are  dependent  on  attention,  and  the  processing  that  occurs  following  the 
sensory  stage  draws  from  a  limited  supply  of  attentional  resources.  Therefore,  an  individual’s  ability  to  attend 
to  stimuli  is  limited  and  influences  how  information  is  perceived,  processed,  and  acted  upon.  Information  that 
does  not  receive  required  attentional  resources  will  not  enter  consciousness.  The  momentary  direction  of  one’s 
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attention  may  be  described  in  terms  of  selective  attention.  The  limits  of  selective  attention  are  realized  when 
unnecessary  elements  of  the  environment  are  selected  for  processing.  Although  selective  attention  is  critical  in 
complex  systems,  most  task  environments  require  operators  to  attend  to  several  sources  of  information 
simultaneously.  A  considerable  amount  of  time  sharing  is  needed,  during  which  the  abilities  to  divide 
attention  and  parallel  process  are  essential.  A  prominent  theory  of  divided  attention  in  human  task 
performance  is  the  multiple  resource  model  proposed  by  Kantowitz  and  Knight  [6]  and  extended  by  Wickens 
[4,7,8].  The  theory  suggests  a  dimensional  system  of  resources  consisting  of  distinct  stages  of  processing 
(encoding,  central  processing,  and  responding),  sensory  modalities  (visual,  auditory),  WM  processing  codes 
(spatial,  verbal),  and  response  modalities  (manual,  vocal).  Each  dimension  is  thought  to  contain  limited 
resources  that  can  be  distributed  between  and  within  tasks. 

With  respect  to  HIP  and  multiple  resource  models,  UMV  operators’  cognitive  workload  can  be  described  as 
the  relationship  between  resource  supply  (HIP  processing  capacity)  and  task  demand.  When  task  demands 
exceed  resource  supply,  an  operator’s  performance  decreases  and  mental  workload  is  perceived.  Of  particular 
importance  for  UMV  operator  interface  design  is  that  the  negative  effects  of  workload  are  triggered  when 
tasks  are  similar  in  the  sensory  modality,  processing  operations,  and/or  responses  given  by  an  individual 
[4,9,10,11].  However,  when  parallel  processing  is  supported  through  the  utilization  of  multiple  modalities 
[4,12]  a  rich  data  environment  can  be  realized,  leading  to  a  system  design  approach  that  successfully 
optimizes  information  processing,  reduces  workload,  and  maintains  enhanced  situational  awareness  in 
dynamic  complex  systems.  This  concept  is  expanded  upon  below  with  respect  to  leveraging  user-centered 
design  concepts  in  developing  UMV  operator  interfaces  to  maximize  information  management  and  individual 
performance  in  C2  environments. 

4.2.1.2  Environmental  Stressors  that  Degrade  Cognitive  Performance 

4. 2. 1.2.1  Impact  of  Provocative  Environments 

One  of  the  benefits  of  operating  in  a  network  centric  environment  is  that  information  requisite  for  effective  C2 
decision  making  can  be  supplied  to  UMV  operators  in  the  field.  However,  the  environment  UMV  operators 
may  be  embedded  in  could  adversely  affect  cognitive  performance,  subsequently  degrading  C2  decision 
making.  In  particular,  environments  such  as  surface  and  ground  operations  that  induce  whole  body  vibration, 
whole  body  motion,  motion  induced  fatigue,  and  motion  sickness  and  associated  negative  after  effects  are  of 
concern.  The  critical  issue  is  that  UMV  operators  may  be  exposed  to  physiological  stressors,  such  as  motion 
sickness,  that  could  degrade  cognitive  and/or  fine  motor  skill  performance  with  a  potentially  damaging  affect 
on  the  attainment  of  a  desired  military  effect. 

With  respect  to  whole  body  vibration  and  motion,  and  motion  induced  fatigue,  the  literature  is  inconsistent 
regarding  their  impact  on  cognitive  C2  task  performance.  It  is  established  that  whole  body  vibration  in  the 
range  from  2  -  12  Hz  can  effect  human  performance  including  decreased  fine  motor  skills,  fatigue,  accident- 
proneness  and  health  hazards  [13,14,15].  However,  the  impact  of  whole  body  vibration  is  not  completely  clear 
because  it  is  a  multi-variate  problem  that  is  further  compounded  by  individual  differences  [16]. 

Research  on  the  impact  of  whole  body  motion  on  human  performance  has  not  revealed  a  direct  relationship 
between  provocative  motion  and  cognitive  performance,  but  it  has  established  a  positive  relationship  with 
motion  sickness  and  degradation  of  fine  motor  skills  [13].  Dobie  [13]  posits  that  dropout  rates  in  whole  body 
motion  studies  may  indicate  that  individuals  are  capable  of  sustaining  a  high  level  of  cognitive  and  general 
efficiency  until  they  remove  themselves  from  the  provocative  sensory  environment.  Research  on  the  ability  of 
individuals  to  complete  C2  tasks  while  underway  in  ground  vehicles  shows  that  vehicle  motion  in  a  variety  of 
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terrain  types  did  not  degrade  cognitive  performance  in  C2  tasks  unless  motion  sickness  resulted,  in  which  case 
cognitive  performance  differences  were  found  between  moving  and  stationary  conditions  [17]. 

Motion  induced  fatigue  is  another  plausible  human  performance  detractor  for  UMV  operators  performing  C2 
tasks  underway  in  ground  and  surface  environments.  The  concern  is  that  motion  induced  fatigue  has  shown  a 
positive  correlation  with  degraded  sense  of  well  being  and  lack  of  motivation  [13].  This  variant  of  motion 
induced  fatigue  called  Sopite  Syndrome,  is  related  to  motion  sickness  and  results  in  individuals  becoming 
unmotivated,  drowsy,  and  experiencing  loss  of  concentration  resulting  in  task  inefficiency  and  accident 
proneness  that  is  not  readily  recognized  by  the  sufferer  or  their  supervisor(s)  [13]. 

The  important  aspect  to  extract  on  whole  body  vibration  and  motion,  and  motion  induced  fatigue  is  that  they 
are  indirect  contributors  to  degraded  cognitive  performance  via  motion  sickness  and  sopite  syndrome 
symptoms.  They  are  also  direct  contributors  to  degraded  physical  abilities  (e.g.,  fine  motor  skills)  that  may  be 
required  to  manipulate  an  autonomous  UMV  C2  user  interface.  In  addition,  it  has  been  shown  that  from  a 
temporal  perspective  the  impact  of  the  aforementioned  environmental  factors  on  cognitive  performance  is 
linked  to  a  step  function  instead  of  a  linear  function  [13].  This  may  explain  why  their  direct  impact  on 
cognitive  performance  has  not  been  explicitly  demonstrated  in  the  empirical  literature  as  a  result  of  short 
exposure  durations  utilized  in  experimental  designs.  The  impact  of  motion  sickness  on  cognitive  performance 
is  discussed  below. 

Motion  sickness  is  another  example  of  a  response  to  provocative  environments  that  causes  diminished 
cognitive  and  motor  performance.  It  is  plausible  that  UMV  operators  performing  C2  tasks  could  be  embedded 
on  surface  and  ground  platforms  where  seasickness  and  vehicle  motion  sickness  are  highly  probable. 
With  respect  to  seasickness  rates  among  naval  warfighters,  Pethybridge  [18]  found  in  the  UK  Navy  that 
10%  to  30%  of  naval  crews  suffered  from  seasickness  during  commonly  experienced  sea  conditions  and 
incidence  rates  of  50%  to  90%  in  high  sea  states.  The  U.S.  Navy’s,  Naval  Medical  Information  Management 
Center  reported  that  from  1980  to  1992,  489,266  new  cases  of  motion  sickness  were  diagnosed  and  106,932 
reoccurrences  were  recorded  [13].  These  figures  represent  a  staggering  loss  of  manpower  and  funds  that 
heighten  the  importance  of  understanding  the  impact  of  seasickness  on  cognitive  performance. 

In  regards  to  the  human  performance  decrements  associated  with  motion  sickness,  the  findings  are 
controversial  depending  on  the  type  of  motion  sickness  (chronic  or  acute)  and  the  performance  metrics  used 
[19,20,21].  However,  findings  on  chronic  motion  sickness  in  military  personnel  induced  by  sustained 
vestibular  stimulation  (e.g.,  motion)  across  multiple  days  revealed  drowsiness,  lethargy,  and  apathy  as  primary 
symptoms  [22].  The  impact  of  these  symptoms  included  participants  being  incapacitated,  refusing  to  perform 
assigned  tasks,  and  spending  a  majority  of  their  time  sleeping  or  lying  down. 

Unfortunately,  the  use  of  pharmacological  interventions  to  mitigate  motion  sickness  introduces  many 
problems.  The  efficacy  and  human  performance  decrements  associated  with  anti-motion  sickness  drugs  is 
subject  to  individual  differences.  However,  the  documented  side  effects  of  these  drugs  have  been  deemed 
unacceptable  for  individuals  making  complex  operational  C2  decisions,  or  in  control  of  sophisticated  or 
potentially  hazardous  equipment  [13].  In  particular,  Cowings  et  al.  [17]  has  shown  that  Promethazine  (an  anti¬ 
motion  sickness  drug),  significantly  degrades  performance  on  cognitive  and  psychomotor  tasks  and  decreases 
alertness. 

While  a  majority  of  the  literature  on  the  impact  of  motion  sickness  on  cognitive  and  fine  motor  skill 
performance  has  been  centered  on  surface  applications  to  drive  ship  hull  design,  some  efforts  [17]  have  looked 
at  C2  operations  in  the  ground  environment  using  the  United  Defense  C2V  M4  vehicle  (a  four  workstation  C2 
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combat  vehicle).  Cowings  et  al.  [17]  used  the  C2V  M4  vehicle  and  C2  tasks  to  measure  performance  on 
cognitive  and  motor  skill  tests,  and  motion  sickness  in  an  array  of  provocative,  but  realistic,  terrains  using 
physiological  sensors  and  subjective  reports.  Seven  of  their  eight  participants  reported  motion  sickness 
symptoms  and  composite  cognitive  and  motor  skill  performance  scores  progressively  declined  during  field 
exercises  in  a  variety  of  terrains.  Four  of  the  participants  indicated  moderate  to  severe  motion  sickness 
symptoms  that  were  not  mitigated  by  short  halts  in  vehicle  movement.  With  respect  to  cognitive  and  fine 
motor  skill  performance,  decrements  in  ability  were  greatest  during  vehicle  motion  and  comparable  to  blood 
alcohol  content  (BAC)  equivalencies  at  or  above  0.08%  in  three  participants  during  movement  and  two 
participants  during  vehicle  halts. 

4.2. 1.2.2  Impact  of  Sleep  Loss  and  Duty  Cycle 

Degradation  of  UMV  operator’s  cognitive  performance  and  information  management  capabilities  in  the 
operational  environment  is  not  limited  to  provocative  motion,  but  may  also  arise  from  sleep  loss  and  duty 
cycle  (e.g.,  shift  schedules).  Sleep  loss  resulting  from  prolonged  operations,  duty  cycle,  and/or  poor  quality 
sleep  due  to  motion  sickness  is  a  valid  area  of  concern  that  may  impact  cognitive  performance  of  UMV 
operators  performing  critical  C2  tasks.  Humans  appear  to  have  a  limited  ability  to  mitigate  the  impacts  of 
sleep  loss,  but  only  when  intrinsic  motivation  is  sufficient,  resulting  in  performance  at  pre-sleep  deprived 
levels  if  wakefulness  does  not  exceed  24  hours  or  the  amount  of  sleep  is  greater  than  50%  of  the  individual’s 
normal  regimen  [23].  The  sleep  loss  research  suggests  that  the  relationship  between  sleep  loss,  performance 
and  motivation  is  not  a  simple  one,  with  motivation  masking  the  effects  of  sleep  loss  on  performance,  but  both 
motivation  and  performance  gradually  being  diminished  by  increasing  sleep  deprivation.  Signal  [24]  and 
Harrison  and  Home  [25]  support  the  notion  of  sleep  loss  ultimately  overcoming  intrinsic  motivation  masking 
effects,  stating  that  a  host  of  cognitive  skills  required  for  decision  making  are  reliant  on  the  prefrontal  region 
of  the  cerebral  cortex,  which  is  affected  by  as  little  as  one  night  of  sleep  loss  [26,27,18,29].  The  skills  that  are 
affected  include:  attending  to  complex  information  while  filtering  out  distractions,  following  a  situation  and 
recognizing  the  need  to  apply  new  strategies,  lateral  thinking  and  innovation,  risk  assessment,  maintaining 
interest,  controlling  mood  and  behavior,  the  ability  to  self  monitor  performance,  and  the  ability  to 
communicate  effectively  [25].  All  of  the  aforementioned  skills  are  key  components  of  effective  C2  decision 
making  in  time  critical  situations,  normal  situations,  and  situations  where  normal  conditions  are  slowly 
degrading. 

In  a  network  centric  environment,  the  ability  to  communicate  and  monitor  performance  is  critical  for  adding 
value  to  the  network’s  information  superiority.  Autonomous  UMV  operators  will  inevitably  be  part  of  a  time 
sensitive  kill  chain  requiring  concise  communication  with  manned  elements,  and  the  ability  to  self-monitor 
performance  is  a  concern  from  an  automation  bias  and  performance  perspective  (automation  bias  occurs  when 
operators  rely  upon  automated  recommendations  and  downplay  contradictory  information)  [30].  With  respect 
to  communication,  the  naturalness  of  speech,  decoding  of  word  meanings,  and  clarity  of  articulation  have 
been  shown  to  degrade  with  sleep  loss  [25].  Harrison  and  Home  [25]  also  noticed  that  sleep  deprivation  lead 
to  increased  confidence  on  ambiguous  tasks  indicating  a  degraded  ability  to  self-monitor  performance,  which 
may  foster  automation  bias  when  interacting  with  automated  decision  aids. 

Another  variable  in  the  cognitive  impact  of  sleep  loss  is  the  influence  of  the  circadian  system  [24]. 
The  combined  effect  of  sleep  loss  and  circadian  rhythm  produces  the  poorest  performance  at  the  circadian 
nadir  [31].  The  implication  being  that  UMV  operators  working  night  shifts  and  experiencing  sleep  loss  are 
more  likely  to  experience  performance  decrements  than  their  counterparts  working  a  day  shift  [32]. 
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4.2. 1.3  The  Need  to  Support  Supervisory  Control  to  Enhance  Information  Management 

The  sections  above  discuss  the  innate  cognitive  capabilities  and  limitations  of  UMV  operators  and  how  they 
could  be  decremented  by  the  operational  environment  (e.g.,  provocative  motion,  sleep  loss,  and  duty  cycles), 
all  of  which  is  critical  for  designing  user  interface  concepts  that  support  information  management  in  the 
complex  C2  domain.  An  additional  information  management  concern  from  a  cognitive  perspective  is  the  need 
for  appropriate  levels  of  automation  to  support  effective  supervisory  control  and  deter  automation  bias. 
Supervisory  control  involves  an  operator  planning  activities  that  are  mediated  by  the  system,  implementing 
the  plan  via  instructional  commands  to  the  system,  monitoring  the  system  to  ensure  the  plan  is  executed, 
intervening  when  the  system  errs  or  needs  assistance,  and  learning  from  the  experience  to  understand  what, 
why,  how,  and  when  the  system  took  the  actions  it  did  to  fulfil  operator  intent  [32].  In  information  rich  NCO 
environments,  effective  information  management  will  be  multi-variate  in  nature  accounting  for  cognitive 
capabilities  and  limitations,  environmental  stressors,  and  appropriate  levels  of  automation  that  facilitate 
cognition  and  supervisory  control  while  avoiding  automation  bias  pitfalls.  All  of  these  aspects  are  essential  for 
supporting  the  use  of  knowledge  based  behaviors  to  complete  C2  tasks  involving  planning,  higher-level 
operation,  and  time  pressured  contingency  interventions  (e.g.,  time  sensitive  targets  and  system  health 
failures).  Discussed  below  are  user-centered  design  operator  interface  concepts  that  show  benefit  for 
information  management  tasks. 
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4.2.2  User-Centric  Information  Management  Concepts  for  Autonomous  UMV  Command 
and  Control 

Given  the  vast  expanse  of  data  that  is  available  to  UMV  operators  in  a  network  centric  environment, 
the  capabilities  and  limitations  of  HIP,  autonomous  UMV  supervisory  control  issues,  and  the  impact  of 
environmental  stressors  on  cognitive  performance,  it  is  essential  that  a  user-centered  design  of  information 
management  user  interface  concepts  is  undertaken.  With  respect  to  information  management  in  a  NCO 
environment  the  challenge  is  not  only  an  abundance  of  data,  but  that  information  is  poorly  organized, 
represented,  and  displayed  to  the  UMV  operator  who  has  to  allocate  attentional  resources  to  an  array  of 
complex  activities.  For  these  reasons,  systems,  displays,  and  technologies  need  to  be  developed  to  aid  in  task 
execution  while  accommodating  cognitive  processing  of  an  operator. 

With  respect  to  current  UMV  operators,  they  are  often  hindered  by  issues  such  as  high  workload  and  reduced 
situational  awareness  (SA)  due  to  the  complexity  of  the  system  and  visual  saturation  [1].  To  mitigate  these 
issues,  UMV  operator  interfaces  need  to  be  designed  with  an  understanding  of  where  the  HIP  bottlenecks 
occur  in  a  task  flow  and  present  information  in  forms  that  are  easily  perceived,  interpreted,  and  responded  to. 
Several  means  for  presenting  C2  information  to  autonomous  UMV  operators  is  discussed  below,  including 
multi-modal  interfaces,  why  to  avoid  3D  visual  displays,  data  fusion,  and  decision  aids. 
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4.2.2.1  Multi-Modal  Interfaces 

Traditional  design  strategy  has  focused  on  visual  interaction  and  this  strategy  is  employed  in  current  UMV 
operator  user  interface  designs  [1].  In  depicting  information  using  spatial  or  graphical  representations, 
visualization  techniques  use  the  human  visual  system  to  facilitate  comparison,  pattern  recognition,  change 
detection,  and  utilize  various  cognitive  skills  by  the  human  perceptual  system.  These  techniques  have  used  a 
relatively  standard  set  of  interaction  paradigms,  leveraging  common  visual  constructs  such  as  windows,  icons, 
menus,  and  direct  manipulation  via  pointing  devices  (WIMPs).  The  pervasiveness  of  these  paradigms  provides 
some  measure  of  their  success,  yet  they  are  limited  when  a  user  is  visually  overloaded.  To  overcome  visual 
saturation,  designers  must  consider  new  design  approaches  that  allow  individuals  to  process  an  optimal 
amount  of  essential  data.  An  approach  discussed  below  leverages  the  auditory  system  in  concert  with  the 
visual  system  because  second  only  to  the  visual  system,  the  auditory  system  is  one  of  the  most  important  and 
highly  utilized  communication  channels.  The  ultimate  goal  of  this  visual-auditory  multi-modal  interface 
concept  is  to  realize  a  holistic  design  that  incorporates  theoretically  sound,  principally  driven  multi-modal 
interface  components  that  achieve  a  genuine  symbiosis  between  user  and  system. 

Instead  of  constraining  UMV  operator  information  management  to  visual  interaction  conventions,  augmenting 
information  via  other  modalities  may  greatly  enhance  HIP.  One  approach  with  great  potential  is  to  leverage 
the  auditory  paradigm  of  Speech,  Earcons,  and  Auditory  Spatial  Signals  (SEAS).  The  main  goal  of  this  multi¬ 
modal  interaction  paradigm  is  augmenting  traditional  visual  interaction  with  auditory  cues  to  substantially 
enhance  cognitive  information  management  capacity.  Multiple  sensory  system  processing  can  substantially 
improve  individual  information  management  capacity  by  enhancing  perception,  augmenting  sensory 
processing,  and  speeding  reaction  time.  Within  the  SEAS  paradigm,  the  design  goal  is  to  present  information 
in  a  modality  that  is  readily  perceived  and  in  a  form  that  is  readily  interpretable.  A  new  class  of  multi-modal 
interactive  systems  would  ensue,  which  would  leverage  available  user  senses,  adapt  to  specific  user’s 
perceptual  and  cognitive  needs,  and  respond  to  such  needs  by  facilitating  intuitive  interaction  with  users. 
System  characteristics  can  then  be  guided  by  the  capabilities  of  human  performance  in  terms  of  sensory 
processing  (e.g.,  is  an  auditory  signal  loud  enough  to  be  heard),  perception  (can  both  visual  and  auditory 
stimuli  be  identified  by  user),  decision  making  (e.g.,  can  a  user’s  working  memory  be  efficiently  used  or  is 
there  cognitive  overload),  and  response  execution  (e.g.,  can  a  user  respond  both  manually  and  vocally).  Based 
on  the  HIP  models  discussed  above,  a  multi-modal  interface  can  be  designed  that  reduces  mental  workload, 
overcomes  HIP  bottlenecks,  and  optimizes  human  performance. 

Effectively  applying  auditory  design  guidelines  to  autonomous  UMV  operator  interfaces  may  lead  to 
improved  C2  task  performance  by  optimally  distributing  information  processing  across  human  sensory 
systems  (i.e.,  visual  and  auditory).  For  example,  Samman  et  al.,  [2]  found  that  participants  processed  nearly  3x 
more  information  when  it  was  distributed  across  various  sensory  systems  (e.g.,  verbal,  tonal,  and  spatial) 
as  compared  to  stimulating  a  single  sense.  In  general,  the  empirical  literature  on  multi-modal  interfaces  suggests 
that  speech,  earcons,  and  auditory  spatial  cues  can  be  processed  simultaneously  with  minimum  interference 
because  such  information  activates  distinct  brain  regions,  thereby  utilizing  different  HIP  resources. 

Realizing  the  full  potential  of  multi-modality  means  not  limiting  the  dimensions  of  HIP  processing  to  the 
verbal-spatial  dichotomies  typically  associated  with  sensory  and  working  memory  processing  codes,  and 
extending  beyond  vocal-manual  response  modalities.  The  findings  of  Samman  et  al.  [2,3]  suggest  that 
leveraging  multi-modal  sensory  systems,  working  memory,  and  response  modalities  promotes  maximum 
information  management  capacity.  Within  this  concept,  the  independence  of  multi-modal  resources  can  be 
leveraged  in  multi-modal  interaction  design  to  strategically  utilize  alternate  resources  at  different  points  in 
operator-system  interaction  to  streamline  an  operator’s  cognitive  load.  Thus,  effectively  designed  multi-modal 
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displays  will  facilitate  perception  such  that  an  operator  searching  a  complex  system  can  accurately  detect 
when,  what,  and  where  in  the  environment  relevant  information  is  located. 

To  validate  the  utility  of  multi-modal  interfaces  for  autonomous  UMV  C2  Samman,  Jones,  Stanney, 
and  Graeber  [4]  implemented  the  interface  design  tenets  bulleted  below  and  adhered  to  speech,  earcons, 
and  spatial  audio  design  guidelines  garnered  from  open  literature. 

•  Speech,  earcons,  and  auditory  spatial  (SEAS)  cues  can  be  processed  simultaneously  with  minimum 
interference  because  such  information  activates  distinct  brain  regions  utilized  by  different  HIP 
resources. 

•  SEAS  should  exploit  human’s  capacity  to  attend  to  a  wide  variety  of  different  sound  dimensions, 
including  location,  pitch,  intensity,  and  semantic  content  to  direct  attention  and  enhance  human 
information  processing. 

•  SEAS  should  design  parallel  task  processing  with  non-overlapping  HIP  resource  demands. 

•  SEAS  suggest  that  an  individual  can  recall  more  in  two  tasks  designed  with  different  types  of 
materials  combined  than  in  a  single  task,  especially  if  the  modalities  or  types  of  representation  are 
very  different. 

•  SEAS  auditory  displays  are  most  appropriate  for  simple  and  short  information  sources. 

•  SEAS  uses  earcons,  auditory  icons,  and  data  auralization  to  semantically  map  information  to 
particular  sound  parameters  (e.g.,  spectral  type,  rhythmic  regularity,  pitch,  timbre,  register,  dynamics) 
or  environmental  sound  cues  to  convey  intended  messages. 

The  multi-modal  autonomous  UMV  operator  interface  Samman  et  al.  [4]  designed  was  empirically  evaluated 
against  a  purely  visual  interface  concept  for  C2  of  UMVs.  Participants  were  asked  to  complete  a  set  of 
primary  tasks  associated  with  a  tactical  situation  display  where  they  captured  radar  images  of  targets  and 
paired  UMVs  with  the  targets  based  on  the  required  weapons  for  effective  prosecution.  Participants  also 
completed  secondary  tasks  associated  with  detection  and  resolution  of  UMV  health  issues.  All  participants 
completed  conditions  requiring  control  of  one,  two,  or  three  groups  of  four  UMVs  where  each  individual 
UMV  required  supervisory  control. 

The  general  pattern  of  results  for  all  of  the  performance  metrics  recorded  support  the  concept  of  integrating 
SEAS  design  tenets  into  C2  displays.  The  objective  of  the  SEAS  auditory  cues  in  this  study  was  a  reduction  in 
operator  attentional  and  visual  perception  bottlenecks,  while  also  assessing  how  many  autonomous  UMVs  a 
single  operator  could  effective  control.  It  was  hypothesized  that  the  integration  of  SEAS  cues  would  reduce 
perceptual  bottlenecks  created  by  a  purely  visual  interface,  thereby  allowing  operators  to  perform  perceptual 
tasks  faster  and  more  consistently. 

Overall,  the  results  suggest  that  the  benefits  of  multi-modal  interfaces  are  realized  as  workload  increases, 
particularly  for  secondary  selective  attention  tasks  [4].  This  may  be  attributed  to  improved  operator  alertness 
levels  and  the  opportunity  to  process  secondary  tasks  and  formulate  an  answer  in  parallel  with  primary  tasks 
when  using  a  multi-modal  SEAS  interface.  In  addition  to  gains  in  primary  and  secondary  task  performance, 
the  data  also  revealed  that  subjective  assessment  of  workload  was  perceived  as  lower  with  the  use  of  the 
multi-modal  SEAS  interface  than  the  baseline  purely  visual  interface  [4]. 
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4.2.2.2  Three  Dimensional  Visualization 

Three  dimensional  display  technologies  come  in  a  variety  of  form  factors  to  suit  an  array  of  applications. 
The  assortment  of  3D  displays  includes  spatially  immersive  displays  (SID)  (e.g.,  domed  simulators,  virtual 
environment  CAVEs),  stereoscopic  head-mounted  displays,  spatially  augmented  reality  (SAR)  displays, 
shutter  goggles,  autostereoscopic  displays,  and  multi-layer  depth  displays.  Intuition  suggests  3D  displays 
should  afford  a  symbiosis  between  an  operator  and  the  local  or  remote  physical  environment  they  are  working 
within,  thereby  enhancing  SA  and  decision  making.  However,  many  studies  have  shown  cognitive 
performance  decrements  for  certain  tasks  when  using  3D  displays  as  compared  to  traditional  2D  displays 
(e.g.,  God’s-eye  or  bird’s-eye  views)  [5, 6, 7, 8].  These  studies  have  shown  that  tasks  requiring  precise 
discrimination  of  relative  depth  or  distance  in  the  z-axis  (orthogonal  to  the  display),  local  surface  orientation, 
relative  curvature,  relative  size,  distance  bisection,  and  co-planarity  are  no  better  with  3D  than  with  well 
designed  2D  displays  and  in  some  cases  significantly  worse  (relative  distance,  local  surface  orientation, 
relative  curvature,  and  global  orientation). 

Three  dimensional  displays  have  also  been  shown  to  be  sub  par  for  information  organization,  a  critical 
component  of  autonomous  UMV  operator  information  management.  Cockburn  and  McKenzie  [9]  examined 
the  efficacy  2D,  2  1/2  D  (receding  incline  plane),  and  physical  and  virtual  3D  models  for  information 
organization  and  retrieval.  The  results  indicated  performance  decrements  with  the  use  of  physical  and  virtual 
3D  information  storage  systems,  suggesting  there  is  limited  use  of  the  third  dimension  in  information 
organization  and  retrieval  tasks. 

4.2.2.3  Data  Fusion 

Data  fusion  is  a  method  for  information  reduction  that  classifies,  correlates  and  filters  sensor  data  to  provide 
UMV  operators  a  more  concise  picture  of  the  battlespace  [10].  Sensor  data  inputs  are  gathered  from  onboard 
(e.g.,  RADAR,  FLIR,  IFF,  etc.)  and  offboard  sources  (e.g.,  JTIDS),  then  analyzed  for  information  quality  and 
proportionally  combined  to  reduce  ambiguity  and  inaccuracy.  Data  fusion  provides  UMV  operators  a 
comprehensive  tactical  picture  that  is  requisite  for  effective  battle  management  by  fusing  the  data  sources  in  a 
manner  that  operators  can  have  high  confidence  in  the  identification  and  position  of  tracks.  The  power  of  data 
fusion  is  seen  in  Figure  4-1  where  a  tactical  picture  is  created  using  un-fused  and  fused  tracks  respectively. 
This  figure  highlights  the  benefits  of  sensor  data  reduction  whereby  a  visual  display  is  created  that  is  easier  to 
interpret  and  it  enhances  decision  making. 


Figure  4-1:  Comparison  of  Tactical  Situation  Displays  with  Unfused  and  Fused  Data. 
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4.2.2A  Decision  Aiding 

As  systems  grow  more  complex,  the  use  of  automation  to  help  operators  complete  time  critical  human 
supervisory  control  tasks  will  be  needed.  However,  Cummings  [11]  notes  it  is  equally  important  to  recognize 
the  essential  role  operators  play  in  supervisory  control  tasks  and  allocate  decision  making  functions  between 
humans  and  computers  accordingly.  In  the  C2  domain,  uncertainty  and  time  pressure  are  elements  in  the 
decision  making  process  and  therefore,  decision  aids  must  provide  autonomous  UMV  operators  the  ability  to 
comprehend  the  battlespace  and  how  various  decisions  may  affect  the  future.  Cummings  and  Guerlain  [12] 
suggest  that  decision  aids  for  C2  applications  must  support  knowledge-based  behaviors  (KBB)  that  require 
complex  cognitive  processing  beyond  application  of  rule  based  behaviors  for  known  situations.  Command  and 
control  tasks  utilize  KBB  because  the  C2  environment  is  unscripted,  open-ended  and  requires  information 
integration  and  evaluation  from  a  multitude  of  sources  to  form  a  decision  in  response  to  events  in  the  battle 
space. 

A  key  to  a  successful  C2  decision  aid  for  a  network  centric  environment  is  the  inclusion  of  human-computer 
interactive  sensitivity  analysis  tools  to  determine  how  warfighter  adjustments  of  decision  variables  could 
impact  an  overall  cost  function  [13].  Allowing  warfighters  to  interact  with  decision  aids  provides  safety 
benefits,  fosters  SA,  and  permits  operators  minor  adjustments  of  computer-generated  solutions  creating  a 
robust  system  capable  of  flexibly  responding  to  uncertain  and  unexpected  events.  Cummings  [13]  notes  that  a 
critical  design  element  of  the  sensitivity  analysis  capability  is  a  data  presentation  user  interface  that  indicates 
the  severity  of  cost  function  change  and  qualitative  information  regarding  the  impact  of  resource  allocation 
shifts  on  the  intended  effect. 

Another  reason  it  is  critical  that  operators  are  kept  in  the  autonomous  UMV  and  decision  aid  supervisory 
control  loops  is  that  UMVs  are  envisioned  to  operate  in  areas  of  uncertainty,  making  them  subject  to 
automation  “brittleness”  [13].  Automation  brittleness  is  the  concept  that  automated  decision-support 
algorithms  are  typically  fixed  in  code  in  initial  design  phases,  and  therefore  unable  to  resolve  unforeseen 
circumstances  [14,15,16].  Higher  levels  of  automation  are  ideal  for  rigid  tasks  that  do  not  require  flexibility  in 
decision  making  and  have  a  low  probability  of  system  failure  [17]  Conversely,  higher  levels  of  automation  are 
not  recommended  for  dynamic  decision  making  environments  like  C2  and  thus  decision  aids  incorporating 
interactive  sensitivity  analysis  are  requisite  because  of  the  risks  and  the  complexity  of  both  the  C2  domain 
system  and  the  inability  of  decision  aids  to  be  perfectly  reliable  [18]. 

An  example  of  a  current  decision  aid  is  the  Associate  concept  where  the  decision  aid  is  able  to  assess  the 
external  and  internal  situation,  make  decisions  based  on  a  common  view  of  the  mission  goals  and  plans  and 
operator  intent,  and  execute  tasks  in  accordance  with  these  goals  [19,20].  As  the  state  of  the  battlespace 
changes  and  data  becomes  available,  data  fusion  provides  a  synthesized  tactical  picture  to  the  autonomous 
UMV  operator  through  the  Information  Manager  (IM).  The  IM  is  responsible  for  deciding  if  and  how  the 
synthesized  data  will  be  presented  to  the  operator.  The  data  is  also  available  to  a  Situation  Assessment 
function  along  with  measures  of  operator  performance  and  intent  gained  by  monitoring  actions  and  comparing 
those  to  pre-determined  plans  and  goals.  Once  a  picture  has  been  developed  of  the  situation  the  Associate  can 
make  decisions  based  on  a  shared  view  of  the  mission  goals  [21].  Through  task  networks  determined  by 
comprehensive  cognitive  task  analysis  of  the  operator  tasks  in  the  battlefield,  the  Associate  can  determine  a 
course  of  action  bounded  by  the  operator’s  intent  and  mission  goals. 

The  Associate  concept  utilizes  the  following  four  levels  of  authority  [21]: 

•  Manual:  The  associate  system  may  never  propose  a  plan  on  its  own.  The  operator  has  full  control  over 
the  plan’s  proposal  and  execution.  The  operator  sends  plans  to  the  associate,  and  the  associate 
responds  like  an  assistant  when  the  plan  has  been  executed. 
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•  Permission:  The  associate  system  may  propose  a  plan,  but  may  not  activate  it  without  explicit 
permission  from  the  operator.  The  associate  sends  the  proposed  plan  to  the  operator.  The  operator  can 
accept  or  reject  the  plan  or  submit  his/her  own.  The  associate  only  executes  plans  that  are  commanded 
by  the  operator.  After  a  pre-programmed  time  with  no  operator  response,  proposals  are  implicitly 
rejected. 

•  Veto:  The  associate  system  may  propose  a  plan  and  may  activate  it  if  the  plan  is  not  explicitly 
rejected  by  the  operator  within  a  given  timeframe.  The  associate  sends  the  proposed  plan  to  the 
operator.  The  operator  can  accept  or  reject  the  plan  or  submit  his/her  own.  However,  if  the  plan  is  not 
explicitly  rejected  or  implicitly  rejected  (because  the  operator  submitted  his  own  plan)  by  the  operator 
within  the  pre-programmed  timeframe,  the  associate  can  execute  the  plan. 

•  Autonomous:  The  associate  system  may  propose  and  activate  a  plan.  The  associate  system  has  full 
control  over  the  plan’s  proposal  and  execution.  Communication  is  limited  to  the  associate  informing 
the  operator  after  execution  is  complete. 

Each  of  these  authority  levels  can  be  assigned  independently  to  the  plans  generated,  giving  the  operator 
flexibility  to  choose  which  portions  of  the  mission  planning  and  tasking  s/he  wants  the  associate  to  undertake. 
Categories  for  assigning  authority  to  the  associate  might  include  information  management  (i.e.,  display 
configuration,  data  transfer),  mission  contingency  planning  (i.e.,  attack  planning),  vehicle  contingency 
planning  (i.e.,  aircraft  and  systems  failure),  and  mission  plan  execution  (i.e.,  attack  scripting). 

The  concept  of  an  Associate  is  beneficial  from  the  perspective  of  lowering  cognitive  demands  utilized  in 
decision  making,  however,  the  level  of  authority  provided  to  the  Associate  has  its  trade  offs  with  regard  to 
supervisory  control,  automation  bias,  and  automation  brittleness.  These  are  all  facets  of  decision  aids  that  need 
future  research,  particularly  within  a  swarming  autonomous  UMV  network  concept. 
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4.2.3  Trust  in  Time  Delayed  Systems 

4.2.3. 1  UMV  and  Time  Delays 

4.2.3. 1.1  Notions  of  Time  Delays 

The  operative  context  of  UMV  is  characterized  by  their  complex  dynamic  environment.  In  such  situations, 
control  is  not  total.  Indeed,  the  operator’s  actions  are  combined  with  the  process  dynamic  and  he  is  not  alone 
in  this  process  [1].  Therefore,  the  environment  in  which  he  is  acting  leads  him  to  update,  according  to  the 
context,  his  mental  representation  of  the  system.  The  operator’s  lack  of  action  is  equal  to  a  change  in  the 
problem  to  solve.  This  kind  of  environment  makes  more  critical  the  management  of  UMVs  because  this 
system  presents  important  time  delays;  cognitive  needs  introduced  by  the  evolving  aspects  of  the  environment 
are  thus  associated  with  latency. 

The  concept  of  time  delay  gathers  a  certain  number  of  subdivisions  (Figure  4-2).  We  can  define  a  response 
delay,  an  information  delay  and  a  feedback  delay.  Each  of  them  has  a  specific  influence  on  the  conduct  of 
robotic  systems. 

Decision  End  of  Effect  Information  Interpretation 

Start  of  action  action  over  effect  of  information 


- ►'  Time 

Response  delay  Information  delay 

< - ► 

Feedback  delay 

Figure  4-2:  Time  Delays. 

A  response  delay  is  the  lag  between  the  action  and  its  effect  on  the  controlled  variable  [1].  It  is  under  the 
influence  not  only  of  automation,  but  also  of  actions  (nature,  time  and  sequence  of  execution).  Indeed, 
from  the  operator’s  point  of  view,  an  information  delay  (time  delayed  information  over  the  effect)  can  be 
added  by  itself  to  the  response  delay.  A  too  long  information  delay  will  lead  to  decrease  for  the  operator  in 
observing  his  own  actions  effect.  Also,  it  makes  the  mental  evaluation  of  the  plan  making  toward  a  specific 
goal  more  difficult  [2].  The  difficulty  of  observing  the  cycle  “information  gathering-diagnosis-action- 
evaluation  of  actions”  all  in  one  makes  the  building  of  mental  representation  of  the  process  evolution  more 
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complicated  [3].  In  this  case,  the  operator  must  hypothesize  the  current  state  of  the  system  in  order  to  make 
decisions  and  corrects  it  according  to  his  results.  However,  a  slick  anticipation  may  be  obviously  not  very 
relevant,  in  situations  where  uncontrolled  factors  can  interfere  with  the  actions  effects.  With  these  difficulties, 
one  found  an  extra  error  production.  Indeed,  Ferrel  [4]  shows  in  a  tele-operation  task,  that  the  presence  of 
information  delay  increases  the  production  of  errors  (59%  of  errors  made  versus  6%  in  non  time  delayed 
conditions). 

Information  over  the  process  given  to  the  operator  is  not  always  synchronised  with  events,  but  can  be 
produced  with  some  time  delay.  The  feedback  delay,  or  latency,  is  thus  the  sum  of  the  response  delay  and  the 
information  delay.  We  notice  an  obvious  deterioration  of  performance  when  time  delay  happens,  because  of 
the  lack  of  synchronization  between  the  operator’s  time  and  the  process  time.  In  practical,  we  note, 
in  a  technological  context,  that  it  is  not  possible  to  reduce  latency  to  zero:  Moreover,  one  will  be  careful  for 
this  latency  to  be  either  constant,  or  equal  for  all  sensorial  feedbacks,  and  inferior  to  a  critical  threshold  [5]. 

The  nervous  system  itself,  have  some  time  delays  in  the  visual-motor  loop.  Nature  has  developed  some 
processes  allowing  us  to  forecast  the  consequences  of  our  actions  and  thus  to  anticipate  the  sensorial  feedback 
up  to  a  certain  level.  With  regard  to  visual  information,  Keele  and  Posner  [6]  have  estimated  this  threshold  at 
200  ms.  In  best  cases,  these  predictive  mechanisms  allow  us  to  compensate  latency  until  a  threshold  of  250 
ms:  any  time  delay  higher  than  this  value  damage  the  performance.  Nevertheless,  over  this  crucial  threshold, 
operators  show  a  certain  form  of  adaptation  to  time  delays  by  using  specific  strategies  allowing  them  to  reach 
a  certain  performance  in  their  activity,  while  accepting  a  lowering  system  dynamics. 


4.2.3.2  Move  and  Wait,  a  Strategy  Facing  Time  Delays 

With  time  delays,  operators  develop  some  strategies  allowing  them  to  fulfil  their  aim.  So,  they  seem  to  adopt 
on  their  own  initiative,  and  spontaneously,  a  “Move  and  Wait”  strategy  (MaW),  i.e.,  they  act,  wait  for  the 
feedback  (in  general  a  visual  one)  before  beginning  a  new  move.  This  strategy  allows  operators  to  limit  the 
incoming  of  unstable  and  unwanted  movements  [7],  but  without  excluding  them.  In  return,  the  task  is 
considered,  by  the  operator,  as  divided  into  various  intermediary  goals  [4];  the  fluidity  of  the  movement  is 
then  particularly  affected.  The  activity  is  no  longer  considered  as  a  whole,  but  like  a  succession  of  given 
actions.  Consequently,  we  notice  a  linear  relation  between  the  completion  time  and  the  intensity  of  time  delay, 
and  this,  even  for  complex  and  time  consuming  tasks  [8];  the  main  effect  being  due  to  the  wait  for  feedbacks. 

Beneath  0.2  s  of  time  delay,  operators  do  not  seem  to  be  obliged  to  use  the  MaW  to  obtain  good  results. 
Beyond  this  value,  not  applying  this  strategy  compels  the  operator  to  provoke  an  oscillation  of  the  system. 
Indeed,  the  late  perception  of  system  responses  leads  the  operator  to  modify  his  instruction  (most  of  the  time 
by  amplifying  it).  The  operator,  noticing  that  this  action  is  over  dimensioned  will  configure  an  opposite 
instruction  which  will  be  in  itself  amplified.  And  thus,  this  process  will  oscillate  and  diverge. 

While  most  of  the  beginners  have  jerky  non-coordinate  and  over  amplified  movements,  training  leads  to 
“smooth”  movement.  In  fact,  the  training  modifies  the  quality  of  concerned  representations,  in  order  to  make 
them  more  flexible  in  the  action  at  a  given  moment.  So,  we  notice  a  smoothing  of  the  imperfections,  with  the 
coming  of  a  strategy  of  delayed  feedbacks:  the  operator  makes  a  certain  number  of  actions  without  waiting  for 
feedbacks  (MoveS  and  Wait).  This  use  of  delayed  feedbacks  shows  some  characteristics: 

•  It  shows  up  after  a  great  number  of  trials. 

•  It  is  rather  the  exception  than  the  rule. 
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•  It’s  only  use  during  in  mastered  activity  (trust  in  the  system). 

•  The  operator  doesn’t  try  to  predict  the  position  or  the  velocity  of  its  actions,  but  only  the  success  or 
the  failure  of  the  action. 

4.2.3.3  Visual  Perception  and  Time  Delays 

The  visual  perception,  through  artificial  sensors,  is  one  of  the  main  means  to  extract  from  the  environment  the 
necessary  elements  to  a  decision-making.  It  allows  evolving  UMV  order.  Yet,  the  operator  must  use  remote 
artificial  vision  systems  and  therefore  through  a  dissociation  between  visual  space  and  action  space. 
This  involves  a  reduction  of  environmental  clues  resulting  from  the  contraction  of  the  visual  field,  and  then 
tends  to  make  the  perception  of  the  action  space  complicated  [9].  The  presence  of  the  screen  provokes  by 
itself  a  change  in  the  control  space  [10].  This  perceptive  activity  can  be  described  by  the  completion  of  two 
distinct  tasks:  the  exploration  of  the  environment  and  the  search  for  targets  within  a  visual  scene  delimited  by 
a  sensor  field.  As  far  as  this  first  task  is  concerned  with  those  feedback  delays,  we  note  a  similar  behaviour  in 
tele-operation  tasks.  So  we  find  again  negative  effect  of  time  delay  in  exploration;  performance  decreases 
through  a  linear  function  with  time  delay.  Consequently,  going  down  to  2.5s  of  time  delay,  we  notice  that 
performance  reduces  by  50%  [11]. 

Added  to  these  problems,  we  found  certain  alterations  of  the  perceptive  mechanisms  implicated  in  visual 
research.  Indeed,  according  to  the  Features  Integration  Theory  [12,13]  various  strategies  can  be  brought  into 
action  with  this  type  of  activity.  However,  with  time  delays,  only  a  serial  features  conjunction  mode  is  chosen 
(sequentially  attention  based  process  of  exploration  which  is  very  expensive  in  cognition  and  not  very  fast) 
[14].  This  choice  will  lead  to  a  compromise  produced  by  a  central  mechanism  of  management  [15]  between 
cost,  performance  and  trust  associated  with  a  given  strategy. 

Being  in  a  downgraded  situation  will  tend  to  select  a  strong  and  statistically  sure  mode  in  order  to  carry  out 
this  task  of  targets  research.  This  choice  is  made  in  spite  of  being  costly.  Indeed,  a  non  detection  becomes 
critical  in  cases  of  strong  time  delays;  the  cost  of  new  research  initialization  becomes  too  important  compared 
with  a  serial  exploration.  Two  mechanisms  can  be  at  the  origin  of  this  behaviour.  First,  fear  of  an  incorrect 
and  ineffective  realization  of  the  task,  creates  certain  anticipation  based  on  previous  experiments.  Thus,  the 
waste  of  time  from  a  non  detection  is  foreseen  by  the  operator.  He  selects,  consciously,  this  mode  of  research 
in  order  to  prevent  this  kind  of  incidents.  Second,  a  cognitive  control/regulation  mechanism  would  carry  out 
during  the  activity  a  cost  evaluation  of  the  tasks  realization.  This  last  would  involve  the  creation  of  a  feeling 
of  difficulty,  with  the  consciousness  of  loosing  control  of  the  situation,  i.e.,  a  “failing  risk  of  the  cognitive 
capacities  because  of  the  saturation  of  the  cognitive  resources”  [16].  This  consciousness  leads  the  operator  to 
select  a  strong  serial  features  conjunction  mode  in  order  to  reduce  this  feeling  of  difficulty. 

From  a  certain  control,  it  becomes  possible  to  release  some  resources  from  the  time  delays  management  to 
prevent  a  “cognitive  overload”,  and  thus  to  restore  a  certain  confidence  in  the  system.  From  this  confidence  is 
restored,  the  acquisition  of  new  research  mode  would  be  possible.  A  redistribution  of  additional  resources 
towards  other  tasks  is  then  possible.  This  implies  that  the  use  of  research  mode  strongly  depends  on  the 
existence  of  a  free  resources  span.  The  reduction  of  this  last  (starting  with  one  second  of  time  delay)  involves 
the  impossibility  of  more  powerful  research  modes  “choice”. 

Time  delays  in  a  multi-tasks  system  involve  quickly  a  cognitive  capacities  overload,  and  thus  induce  a  change 
of  strategies  used  in  the  realization  of  the  activity.  We  note  the  use  of  MaW,  with  regard  to  the  exploration  of 
the  environment,  or  the  inhibition  of  powerful  perceptive  mechanisms  of  visual  research.  These  facts  bring  the 
operator  to  wonder  about  his  self-confidence  and  the  trust  in  the  system.  This  questioning  is  the  result  of 
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uncertainties  due  to  system  time  delays,  involving  difficulties  of  representation  and  thus  of  understanding  the 
situation. 
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4.2.4  Uninhabited  Military  Vehicles  and  Trust 

4.2.4.1  Time  Delays  and  Representations 

The  UMV  system  with  all  the  inherent  time  delays  creates  spatial  and  temporal  uncertainties  for  the  operators. 
This  means  that  real  time  reaction  is  not  possible  [1].  Keren  and  Roelofsma  [2]  suggest  that  uncertainty  and 
time  delays  affect  the  decisions  through  only  one  common  dimension.  Time  delays  eliminate  the  effect  of 
certainty  such  as  uncertainty  would  do,  because  time  delays  carries  part  of  risk,  which  leads  to  uncertainty. 
The  problem  arises  from  dissociation  between  the  operator’s  universe  and  the  UMV  universe.  The  more 
important  this  variation  is,  the  more  it  is  difficult  to  synchronize  process  time  and  operator’s  time. 
This  involves  great  difficulties  in  answers  adjustment,  which  must  be  relevant  with  the  time  sequences  of  the 
environment.  The  operator  encounter  two  linked  difficulties  in  this  process,  to  adapt  the  production  of 
commands  in  due  time,  and  to  build  and  maintain  a  representation  of  the  system  in  a  temporal  and  implicitly 
spatial  scale  because  of  the  environment  dynamics.  The  cognitive  requirements  related  to  the  UMV 
management  are  fulfilled  by  the  creation  of  an  internal  representation  of  the  system,  its  evolution,  and  the 
action  plans.  This  represents  more  or  less  the  reality  and  depends  on  the  nature  of  the  external  representations 
[3],  which  can  be  considered  as  the  primary  support  of  the  activity.  Their  informational  levels  are  directly 
associated  with  the  quality  of  the  operator’s  internal  representation.  Thus,  the  simpler  the  operator’s  system 
model  is,  the  closer  the  feedbacks  must  be  to  reach  an  acceptable  level  of  performance.  The  more  distant 
feedbacks  are,  the  less  the  operations  on  these  mental  representations  are  able  to  work,  and  the  more  the 
model  needs  other  regulations  such  as  anticipations.  Paradoxically,  anticipations  require  relatively  powerful 
system  of  internal  and  external  representation. 

Time  delays  in  a  system  leads  to  some  difficulties  of  building  and  maintain  relevant  mental  representations 
with  respect  to  a  given  objective. 


4.2.4.2  Representation  and  Understanding 

The  representation  of  the  world  allows  its  understanding.  However,  the  knowledge  by  itself  of  the  system  in 
its  environment  cannot  fully  satisfy  understanding.  Indeed,  understanding  is  not  a  property  of  environments; 
there  is  no  understanding  without  an  intention  which  directs  the  analysis  and  building  of  relations  on  a  limited 
part  of  the  environment  components  [4].  Thus,  intention  and  understanding  are  closely  linked:  understanding  a 
situation  consists  in  building  a  coherent  representation  of  the  world  for  the  goal  to  achieve  and,  to  update  it  to 
satisfy  the  evolving  requirements  of  this  universe.  The  difficulty  of  understanding  the  situation  comes  from 
the  weak  quality  of  the  system  representation.  Creating  consistency  between  past,  current,  and  future  events 
and  the  operator’s  intentions  is  complex.  The  lack  of  predictability  leads  to  a  restriction  of  anticipations, 
a  short-term  control,  a  permanent  high  workload  [4,5,6],  and  an  increased  need  for  confidence  [7].  Problems 
related  to  the  building  of  an  operative  system  representation  in  an  environment,  leads  to  a  progressive  loss  for 
the  operator’s  understanding  over  his  controlled/supervised  system. 

4.2.4.3  Understanding  and  Trust 

In  the  case  of  UMV,  the  symbolic  intermediaries  create,  for  the  operator,  a  certain  representation  of  the  world 
[8].  This  synthetic  view  of  the  reality  raises  the  question  of  trust  in  the  system.  Indeed,  information  for  the 
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operator  presents  some  specific  deformations  of  the  vehicle  environment  (be  it  wanted  or  not).  Then,  the 
understanding  of  the  situation  depends  of  the  consistency,  towards  a  given  intention,  of  the  external 
representation  of  the  world  provided  by  the  system,  and  the  world  itself.  The  ease  and  stability  of  this  relation 
are  directly  related  to  the  quality  and  the  speed  of  trust  acquisition.  Well,  the  temporal  gap  between  action  and 
his  effect  tends  to  increase  the  uncertainty  and  then  the  need  for  trust  [9,10]. 

Trust  can  be  defined  as  “the  attitude  that  an  agent  will  help  achieves  an  individual’s  goals  in  a  situation 
characterized  by  uncertainty  and  vulnerability”  [11].  In  the  case  of  a  decision  aid,  trust  is  “...  the  extent  to 
which  a  user  is  confident  in,  and  willing  to  act  on  the  basis  of  the  recommendations,  actions,  and  decisions  of 
an  artificially  intelligent  agent”  [12].  This  can  be  determined  by  a  stake  with  a  probability  of  an  adverse 
outcome  [13].  Two  different  notions  can  define  trust:  self  confidence  (egocentric)  and  trust  in  the  system 
(exocentric).  It  is  their  ratio  which  indicates  the  level  of  acceptable  risk  (internal  and  external)  for  the 
operator.  Then,  we  can  consider  trust  like  a  risk  which  is  accepted  to  prevent  other  risks  [4]. 

4. 2. 4. 3.1  Trust :  a  Regulator  of  the  Cognitive  Compromise 

The  operator  confidence  seems  to  evolve  with  the  mechanisms  of  acquisition  and  particularization  of  his  know¬ 
how  [14];  the  ease  for  detecting  the  behaviour  of  the  system  and  its  supporting  state  indicators  increase  this 
confidence  [15].  When  confident,  the  operator  tends  to  automate  certain  specific  know-how  with  regards  of  his 
task.  This  behaviour,  associated  with  a  certain  simplification  and  loss  of  flexibility  of  the  system  management, 
structures  and  facilitates  the  causal  analysis.  The  operator  makes  “more”  (in  term  of  performance)  with  “less” 
(in  term  of  knowledge)  [4].  However,  the  flexibility  of  adaptation  to  the  context,  resulting  from  confidence, 
cannot  be  directly  associated  with  the  know-how.  Indeed,  confidence  is  more  related  to  an  “ability  of 
management”  than  with  a  know-how  control.  The  degree  of  confidence  in  a  system  allows  the  operator  to 
explore  a  more  or  less  wide  field  of  his  cognitive  compromise  with  respect  to  certain  know-how,  and  then  allows 
him  to  increase  his  possibility  of  actions  on  the  situation.  This  flexibility  of  the  cognitive  compromise  authorizes, 
from  time  to  time,  the  operator  to  take  more  risks  (a  lower  control)  in  exchange  of  a  greater  adaptability  in  his 
task.  This  risky  behaviour  obeys  to  a  principle  of  homeostasis,  in  accordance  with  the  modification  of  the 
environment  [16].  Then,  we  can  define  a  judgement  of  confidence  like  a  metacognitive  activity  [17,18,19]; 
confidence  is  a  regulator  of  the  operator’s  level  of  control.  In  the  case  of  a  low  confidence,  the  operator  tends  to 
select  a  conscious  and  logical  control  mode  based  on  his  metaknowledge,  which  is  associated  with  a  great 
workload.  On  the  contrary,  a  high  confidence  involves  a  reduction  of  the  level  of  control,  with  an  increasing 
production  of  routine  errors  in  the  case  of  overconfidence  [4,13]. 

4. 2. 4. 3. 2  Trust  and  Automation 

Confidence  of  an  operator  in  an  automated  system  determines  its  behaviour  toward  the  system  [20,11,21,7]. 
Lee  and  Moray  [22]  suggest  that  operators  use  automatic  modes  in  accordance  to  their  previous  level  of 
confidence,  the  probability  of  failures  and  their  skills  in  manual  control.  Decisions  to  switch  over  from  one 
mode  of  command  to  another  can  be  predicted  by  the  ratio  between  the  level  of  confidence  in  the  system  and 
the  operator’s  self-confidence  [23,24,25].  Therefore,  the  choice  of  a  manual  control  shows  a  confidence  in  the 
manual  mode  better  than  in  automation.  This  choice  is  made  on  a  subjective  evaluation  of  the  reliability  of  the 
system  and  on  the  own  skills  of  the  operator.  Indeed,  an  operator  will  not  use  a  reliable  automatic  system  if  he 
does  not  trust  it.  However,  in  the  case  of  system  which  presents  failures,  the  operator  can  judge  his  own 
capacity  to  manage  it  manually  less  reliable  and  riskier  than  in  an  automatic  mode  due  to  workload  increase 
[26,27].  Then,  the  operator  prefers  to  switch  to  an  automatic  mode  while  reinforcing  his  level  of  supervision. 
This  can  be  confirmed  by  the  fact  that  the  level  of  detection  of  failures  by  the  operator  varies  conversely  with 
the  reliability  of  automation  [28]. 
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In  the  first  interaction,  we  note,  without  failures,  a  quick  raise  of  a(n)  (over)confidence  of  the  operator  towards 
the  system  [29]  especially  in  the  case  where  decisions  are  taken  from  visual  information  [17].  This  confidence 
remains  until  important  failures  happen.  Indeed,  occasional  failures  do  not  induce  a  change  of  the  control 
mode  as  long  as  theses  are  bearable  for  the  operator  with  a  lower  cost  in  mental  workload.  Nevertheless, 
Lee  and  Moray  [22]  have  shown  that  whatever  problems  encountered  confidence  in  automation  settles  slower 
than  it  decreases. 

We  note  that  certain  researchers  consider  that  man,  in  his  interaction  with  an  artificial  system,  reproduces  the 
same  behaviours  of  management  of  trust  as  those  observed  in  social  situations  between  individuals 
[30,31,32,33]  and  this  in  spite  of  the  fact  that  technological  artefacts  do  not  have  motivations.  These 
researchers  talk  even  about  “personality”  of  the  system  [34]  and  about  its  links  with  the  operator’s  personality. 

4.2.4.4  Conclusions 

The  difficulties  (exploration  of  the  environment  and  visual  searches)  associated  with  time  delays  lead  the 
operator  to  reconsider  his  confidence  about  the  system,  often  with  a  negative  view.  It’s  not  in  favour  of  a  strict 
control  by  the  operator,  but  rather  for  supervision  and  task  sharing  between  the  operator  and  the  system. 
Nevertheless,  we  have  to  be  careful  to  not  impoverish  the  operator’s  activity  and  then  consequently  his 
understanding  of  the  process.  In  the  context  of  a  hermetic  system  (for  the  operator),  the  activity  of  supervision 
tends  to  decrease.  Indeed,  the  use  of  automation  and  then  the  difficulty  of  control  are  associated,  in  a  timely 
manner,  with  a  reduction  of  skills  and  the  ability  to  supervise  the  system  [35].  Recovery  of  errors  by  the 
operator  is  then  particularly  affected. 

With  these  problems  must  be  added  risks  about  the  implication  of  the  operator  towards  the  UMV  in  his 
control/supervision.  Indeed,  spatial  and  temporal  distance  between  the  command  station  and  the  aircraft  lead 
the  operator  in  a  no  interpretation  status  of  the  bonds  between  these  two  entities.  Thus,  the  intermediary  space 
between  these  two  entities  is  considered  like  a  black  box  where  the  outputs  do  not  seem  to  be  related  to  the 
inputs  of  the  operator.  This  space  tends  to  be  an  independent  operative  intermediary  between  the  operator  and 
the  UMV.  This  creates  a  gap  between  these  two  entities  which  move  in  two  different  spatial  ant  temporal 
universes,  and  then  leads  the  operator  to  work  only  in  the  universe  which  can  be  directly  controlled.  Then,  the 
operator  will  consider  his  task  only  toward  an  external  artificial  representation  of  the  UMV  (the  screen  in  the 
command  station)  and  not  in  relation  to  a  real,  and  distant,  massive  aircraft. 
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4.2.5  Future  Research 

As  knowledge,  technology,  and  autonomous  UMV  system  capabilities  progress,  HSI  research  efforts  must 

also  continue  to  address  user-centered  design  aspects  of  information  management  for  C2.  Five  pertinent  areas 

of  enabling  technology  research  are  discussed  below,  including,  multi-modal  user  interfaces,  augmented 

cognition,  collaborative  tools,  chat  applications,  and  supervisory  control  of  autonomous  UMV  swarms. 

See  the  “Advanced  UMV  Operator  Interfaces”  chapter  within  this  publication  for  additional  design  concepts. 
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4.2.5.1  Multi-Modal  Interfaces 

Continued  multi-modal  interface  research  is  needed  to  further  understand  HIP  and  how  best  to  leverage 
cognitive  resource  and  attention  pools  utilized  in  processing  sensory  inputs,  decision  making,  and  response 
execution.  The  concern  is  that  HIP  benefits  from  presenting  information  multi-modally  could  be  tempered  if 
the  costs  for  modality  coupling  and  switching  are  high.  Another  vein  of  research  is  an  examination  of  the 
potential  benefits  of  multi-modal  interfaces  to  mitigate  the  effects  of  visually  induced  motion  sickness  in 
provocative  environments  while  allowing  the  operator  to  remain  effective  in  performing  C2  tasks.  Multi¬ 
modal  user  interface  research  should  also  investigate  the  potential  for  supporting  reduced  manning  concepts 
and  personnel  selection  criteria  for  effective  use  of  multi-modal  systems.  Finally,  in  conjunction  with  a  deeper 
understanding  of  how  multi-modal  interfaces  leverage  HIP  concepts,  research  should  also  be  undertaken  to 
compliment  the  augmented  cognition  concept  discussed  below.  In  particular,  the  creation  of  modality  based 
user  interface  augmentation  guidelines  for  cognitive  bottleneck  resolution  in  response  to  real  time  assessment 
of  operator  workload  and  the  HIP  resources  overburdened. 

4.2.5.2  Augmented  Cognition 

Augmented  cognition  is  a  neuroergonomics  enabling  technology  concept  that  leverages  HIP  models, 
user  interface  augmentation  concepts  founded  in  HIP  theory  (e.g.,  multi-modal  user  interfaces), 
and  physiological  sensor  technologies.  The  goal  of  augmented  cognition  is  to  develop  a  cognitive  feedback 
loop  between  operator  and  system  that  allows  the  system  to  sense  when  an  operator  is  experiencing 
unacceptable  levels  of  cognitive  workload,  mental  fatigue,  and/or  stress.  When  the  aforementioned 
contributors  to  cognitive  performance  decrement  are  detected,  an  augmentation  manager  appropriately  alters 
the  user  interface  or  information  presentation  modality  to  mitigate  cognitive  performance  decrements. 

From  an  applied  perspective,  the  goal  of  augmented  cognition  is  to  increase  the  amount  of  information  that 
operators  can  process  and  utilize  in  decision  making,  reduce  manpower  requirements,  and  improve  selective 
attention  during  stressful  battlefield  conditions.  These  applied  goals  are  related  to  autonomous  UMV  C2  by 
directly  impacting  the  number  of  automated  UMVs  an  operator  can  control  and  providing  an  effective  means 
for  enhancing  cognitive  processing  and  information  management.  Augmented  cognition  research  efforts 
relating  to  autonomous  UMV  C2  [1]  have  utilized  a  suite  of  physiological  sensors  (pupil  size,  EEG,  ECG, 
EOG,  EMG,  and  functional  near  infrared  (fNIR))  and  statistical  process  control  techniques  to  assess  an 
operator’s  cognitive  state,  while  employing  an  array  of  user  interface  augmentations  to  mitigate  high  cognitive 
workload  contributors.  Barker  [1]  conducted  a  series  of  experiments  to  address  cognitive  bottlenecks  in 
working  memory,  executive  function,  sensory  input,  attention,  and  response  generation.  The  experiment 
results  demonstrate  that  aiding  from  closed-loop  augmented  cognition  significantly  improves  autonomous 
UMV  operator  performance  on  both  primary  and  secondary  C2  tasks.  The  primary  tasks  that  showed 
improvement  were  associated  with  ingress  and  attack  phases  for  suppression  of  enemy  air  defences  (SEAD) 
mission  where  the  operator  was  controlling  one,  two,  or  three  strike  packages  consisting  of  four  UMVs. 
The  secondary  tasks  that  showed  improvement  were  detecting  and  responding  to  vehicle  health  alerts. 

While  research  in  the  domain  of  augmented  cognition  is  based  on  a  solid  theoretical  foundation,  substantial 
work  remains  to  validate  the  efficacy  of  the  concept  at  both  basic  and  applied  levels.  Future  research  is  also 
required  to  improve  physiological  sensor  capabilities,  packaging,  and  optimal  placement.  In  conjunction  with 
sensor  development  is  the  improvement  of  techniques  for  processing  the  sensor  data  and  selection  of 
appropriate  user  interface  augmentations.  Augmented  cognition  is  a  promising  concept  that  could  be  an 
enabling  technology  breakthrough  for  autonomous  UMV  C2  as  the  associated  body  of  knowledge  continues  to 
expand  and  mature. 
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4.2.5.3  Chat  Applications 

The  use  of  chat  applications  and  instant  messaging  were  discussed  above  as  potential  tools  to  enhance  a 
variety  of  collaborative  C2  scenarios.  They  are  revisited  in  this  section  because  they  may  result  in  an 
unexpected  consequence  of  the  convergence  of  network  centric  technical  capabilities  and  societal  trends. 
Cummings  and  Guerlain  [2]  used  chat  during  a  decision  aid  study  as  an  operationally  valid  secondary  task  to 
measure  workload.  Unexpectedly,  they  found  that  the  chat  application  and  the  information  contained  in  the 
chat  messages  dominated  operator  attention  allocation  while  performing  time  sensitive  retargeting  of  tactical 
tomahawk  missiles.  Cummings  and  Guerlain’s  [2]  intent  was  to  create  a  realistic  secondary  task  that  required 
spatial  reasoning  skills  in  a  chat  application  that  was  familiar  to  naval  personnel.  The  chat  application 
conveyed  messages  containing  basic  information  about  missile  status,  instructions  for  action,  or  queries  for 
information  from  superiors  about  past,  present,  and  future  elements  of  missile  and  target  status.  Interestingly, 
Cummings  and  Guerlain’s  [2,3]  found  that  participants  became  fixated  on  the  chat  application  despite  explicit 
instructions  that  time  critical  retargeting  was  the  highest  priority  task  and  answering  chat  queries  was  the 
lowest  priority  task.  In  addition,  participants  were  told  that  incoming  chat  messages  were  generated  by  a 
scripted  computer  program  and  their  responses  were  not  being  read  by  a  human.  In  spite  of  stressing 
instructions  to  only  attend  to  chat  when  no  other  tasks  were  occurring,  many  participants  responded  to  chat 
queries  before  attending  to  the  more  pressing  time  sensitive  target  problems.  Cummings  and  Guerlain  [2,3] 
suggest  that  the  over-attention  to  the  chat  application  degraded  the  overall  task  performance  of  some 
participants,  which  could  have  costly  operational  consequences. 

The  findings  of  Cummings  and  Guerlain  [2,3]  indicate  future  research  may  be  needed  to  understand  the 
predilections  of  an  e-enabled  warfighter  population  that  regularly  uses  social  connectedness  applications,  such 
as  chat  and  IM,  in  their  work  and  personal  lives.  Inevitably,  current  and  future  warfighters  responsible  for  C2 
of  UMVs  will  bring  with  them  generational  culture  biases  where  compelling  personal  connections  are 
established  via  e-enabled  medium.  As  network  centric  technical  capabilities  evolve  and  incorporate  common 
place  commercial  applications,  future  research  should  be  directed  at  understanding  the  societal  impact  of  these 
technologies  on  the  military  community  and  the  possible  impact  on  training,  tactics,  and  procedures  as  well  as 
warfighter  performance. 

4.2.5.4  Decision  Aiding  for  Autonomous  UMV  Swarm  Control 

Autonomous  swarming  UMV  networks  will  create  new  aspects  of  operator  decision  making  complexity  in  C2. 
One  of  the  primary  advantages  of  autonomous  UMV  swarming  networks  is  the  ability  to  process  large 
amounts  of  sensor  data  in  short  periods  of  time  to  optimally  achieve  an  intended  effect.  Humans  will  continue 
to  be  needed  in  autonomous  swarm  networks  to  ensure  safety  and  monitor  progress  toward  an  intended  effect 
as  part  of  the  supervisory  control  and  decision  making  loops.  However,  Cummings  [4]  notes  that  because  of 
the  revolutionary  nature  of  swarming  technology,  futuristic  operator  interaction  with  complex  autonomous 
UMV  swarms  is  not  well  understood,  and  more  research  emphasis  needs  to  be  placed  on  operator 
requirements,  strengths,  and  limitations  for  supervisory  control.  Cummings  [4]  suggests  that  future  research  in 
this  domain  should  include  an  investigation  of  the  interaction  of  increasing  vehicle  autonomy  on  human 
supervisory  control,  the  effect  of  increased  levels  of  automation  in  decision-making,  and  how  situation 
awareness  is  affected  by  increasing  levels  of  vehicle  autonomy  and  automated  decision  making.  The  crux  of 
the  future  research  issue  is  to  devise  an  autonomous  UMV  swarm  -  operator  feedback  loop  that  allows  the 
operator  to  understand  what,  how,  and  why  a  swarm  behaves  like  it  does  [4]. 

While  current  and  past  research  has  focused  on  determining  levels  of  automation  that  promote  effective 
operator-system  interaction  [5,6,7,8,9,10],  little  has  focused  on  the  C2  domain,  and  virtually  no  research  has 
been  done  on  operator-autonomous  swarm  interaction.  Cummings  [4]  notes  that  with  the  creation  of 
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autonomous  UMV  swarm  networks  that  communicate  with  both  humans  and  each  other,  the  supervisory 
control  problem  space  increases  in  scope  and  complexity.  From  a  future  operator  decision  aid  design 
perspective,  the  implication  is  that  it  will  be  paramount  to  determine  the  impact  of  automation  levels  for 
decision  making,  the  effects  of  different  collaboration  levels  between  UMVs,  the  interactions  between  the  two 
automated  systems  (i.e.,  decision  aids  and  swarming  UMV  networks),  and  how  best  to  represent  to  the 
operator  multi-objective  cost  functions  that  should  either  be  minimized  or  maximized  [4].  From  a  high  level 
perspective,  future  research  on  supervisory  control  and  operator  decision  aids  for  autonomous  UMV  swarms 
should  focus  on  the  impact  of  increasing  vehicle  autonomy  on  supervisory  control,  the  effect  of  increased 
levels  of  automation  in  decision  making,  and  how  situation  awareness  is  affected  by  the  increasing  levels  of 
UMV  autonomy  and  decision  making  automation  Cummings  [4]. 
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4.3  MIGRATION  OF  OPERATOR  CONTROL:  HUMAN  FACTORS  AND 
TEAMING  ISSUES 

4.3.1  Background 

As  noted  by  Gawron  [1]  specifically  with  regards  to  uninhabited  aerial  vehicles  (UAVs),  the  advent  of 
uninhabited  military  vehicles  (UMVs)  has  created  a  host  of  new  human  factors  challenges  which  arise 
primarily  because  the  vehicle  and  the  operator  are  no  longer  necessarily  co-located  [2].  Perhaps  one  of  the 
most  unique  of  these  UMV  specific  human  factors  challenges  concerns  migration  of  operator  control.  While 
migration  implies  movement,  the  construct  of  control  migration  used  in  this  work  will  be  similar  to  that 
described  by  Kahne  [3]  and  includes  changes  in  the  locus  of  control  within  functional,  temporal,  or  physical 
domains.  For  instance,  in  current  long  endurance  UAV  operations,  control  may  be  transferred  between 
operators  in  a  control  station  (e.g.,  crew  changeover),  between  control  stations  (e.g.,  vehicle  handoff), 
or  among  members  of  a  crew  (e.g.,  task  execution)  [4].  Although  migration  of  operator  control  has  been  a 
factor  in  several  UAV  mishaps  [5,6],  there  are  currently  no  relevant  studies  in  the  literature  addressing  this 
issue  in  either  UAVs  [4]  or  other  UMV  systems.  For  autonomous  UMVs  requiring  only  supervisory  control, 
studies  of  air  traffic  control  (ATC)  can  serve  as  an  analog  supervisory  control  domain  [7].  However,  while 
ATC  issues  instructions  to  aircraft,  ATC’s  role  is  ultimately  advisory  rather  than  mandatory  since  legal 
responsibility  for  the  safety  of  the  aircraft  and  its  occupants  rests  with  the  pilot  [8].  Thus,  this  section  will 
necessarily  be  more  of  a  theoretical  discussion  of  migration  of  operator  control  rather  than  a  review  of 
empirical  evidence.  In  particular,  migration  of  operator  control  in  UMVs  will  be  explored  as  a  novel 
application  of  the  fields  of  team  processes  and  team  communications.  First,  the  current  state  of  knowledge 
regarding  migration  of  operator  control  in  the  most  mature  UMV  technology  (e.g.,  UAVs)  will  be  reviewed 
with  the  understanding  there  are  logical  applications  to  future  ground  or  maritime  UMVs  as  well.  This  will  be 
followed  by  discussions  of  the  potential  reasons  for  and  the  advantages  and  disadvantages  of  migrating 
operator  control,  the  impact  of  migrating  operator  control  on  team  dynamics,  important  issues  which  need  to 
be  resolved,  and  potential  strategies  to  address  or  mitigate  those  issues. 
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4.3.2  Migration  of  UAV  Operator  Control 
4.3.2.1  Levels  of  Control 

Standardization  Agreement  (STANAG)  4586,  Standard  Interfaces  of  the  UAV  Control  System  for  North 
Atlantic  Treaty  Organisation  (NATO)  Interoperability,  defines  five  different  Levels  of  Interoperability  (LOIs) 
or  degrees  of  control  for  UAVs  (A.  Kirschbaum,  personal  communication,  August  17,  2005): 

Level  1 :  Reception  and  transmission  of  secondary  imagery  or  data. 

Level  2:  Reception  of  imagery  or  data  directly  from  the  UAV. 

Level  3:  Control  of  the  UAV  payload. 

Level  4:  Control  of  the  UAV,  without  takeoff  and  landing. 

Level  5:  Full  function  and  control  of  the  UAV  to  include  takeoff  and  landing. 

Control  complexity  increases  from  level  1  to  level  5  with  each  subsequent  level  including  the  capabilities  of 
the  former  level(s)  (Figure  4-3).  Implicit  in  this  control  taxonomy  is  the  understanding  that  a  variety  of 
transfers  between  LOIs  may  be  necessary  during  a  single  UAV  mission.  STANAG  4586  describes  these  LOIs 
without  reference  to  a  specific  UAV  category.  Nor  does  the  standard  clearly  state  if  these  LOIs  apply  only  to 
the  dedicated  UAV  control  station  or  if  they  also  apply  to  downstream  users  of  the  information  derived 
directly  from  the  UAV.  Certainly  allowing  a  user  more  control  of  the  system  has  the  potential  to  enhance 
mission  execution  by  decreasing  the  number  of  intermediary  personnel.  However,  the  higher  the  LOI, 
the  more  costly  the  equipment  and  the  more  specialized  the  training  required  [1]. 
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Figure  4-3:  Levels  of  UAV  Interoperability  and  the  Respective  Communication  Links. 
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4.3.2.2  Types  of  Control  Migration 

4. 3. 2. 2.1  Changeover 

A  changeover  is  the  migration  of  vehicle  or  payload  control  from  one  (group  of)  operator(s)  to  another 
(group  of)  operator(s)  at  the  same  location.  It  is  possible  during  a  changeover  to  have  a  face-to-face  debrief. 
Also,  since  the  same  equipment  is  used,  there  is  no  need  for  system  changes  or  data  transmission  link 
reconnections.  Additionally,  a  changeover  will  generally  not  require  coordination  with  ATC  or  other  external 
command  and  control  (C2)  agencies.  Types  of  changeovers  include: 

•  Time  transfer:  Time  transfer  implies  the  operators  are  identical  in  skill  and  function  and  control  is 
transferred  because  the  endurance  of  the  vehicle  exceeds  that  of  the  operator(s).  Very  long  UAV 
missions  may  require  several  time  transfers. 

•  Function  transfer:  Function  transfer  implies  the  operators  must  accomplish  different  tasks  during  the 
same  mission,  possibly  in  another  system-mode  or  even  at  a  different  part  of  the  system.  For  example, 
an  operator  may  merely  perform  navigation  and  safety  monitoring  tasks  in  coordination  with  ATC 
during  the  ferry  to  the  theatre  of  operations  (TOO),  but  once  in  the  TOO,  may  change  to  a  more 
payload  driven  way  of  operations  (Figure  4-4).  This  form  of  transfer  can  be  combined  with  a 
changeover. 

•  Skill  transfer:  Skill  transfer  implies  the  operators  are  trained  differently  and  the  transfer  is  required  as 
the  vehicle  performs  different  tasks.  This  kind  of  transfer  may  be  performed  when  less  experienced 
operators  operate  the  vehicle  or  payload  for  simple  tasks  while  more  skilled  operators  take  over  for 
complex  tasks. 


EN-ROUTE 


THEATRE  OF 


THEATRE  OF 
OPERATIONS 


PAYLOAD  CONTROL  2 


Figure  4-4:  Functional  Transfer  and  Time  Transfer  of  the  UAV  during  Different  Mission  Phases. 
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A  handoff  or  handover  involves  the  migration  of  vehicle  or  payload  control  from  one  location  to  another 
location.  Often  the  term  handover  is  used  when  the  transfer  is  performed  between  two  similar  control  systems 
(e.g.,  two  identical  control  stations).  In  contrast,  the  term  handoff  is  used  when  the  transfer  is  performed 
between  two  dissimilar  control  systems  (e.g.,  changes  between  LOIs). 

4. 3. 2. 2. 2  Vehicle  versus  Payload 

Both  a  changeover  and  a  handoff  may  be  used  to  transfer  control  of  both  the  vehicle  and  payload  or  solely  the 
payload.  A  vehicle  transfer  implies  operators  transfer  control  of  the  vehicle  to  include  the  following: 
a)  vehicle  command  and  control,  b)  navigation,  c)  voice  communications,  d)  vehicle  safety  and  emergency 
responsibility,  e)  payload  control,  and  f)  data  transmission  link  control  and  monitoring.  In  contrast,  a  payload- 
only  transfer  implies  operators  retain  control  of  the  vehicle  (items  a-d)  and  only  transfer  control  of  the 
payload.  Payload-only  transfer  appears  less  complex  than  vehicle  transfer  because  overall  responsibility  for 
the  vehicle  does  not  change.  However,  if  a  large  area  must  be  covered  or  objects  of  interest  are  mobile,  a  high 
degree  of  coordination  is  required  between  the  vehicle  and  payload  operators  in  order  to  keep  the  UMV  path 
within  sensor  constraints.  This  may  significantly  increase  the  workload  of  both  operators  and  have  a  negative 
effect  on  the  operators’  situational  awareness  (SA). 

4.3.2.3  References 

[1]  Theunissen,  E.,  Goossens,  A.A.H.E.,  Bleeker,  O.F.  and  Koeners,  G.J.M.  (2004,  August).  UAV  mission 
management  functions  to  support  integration  in  a  strategic  and  tactical  ATC  and  C2  environment 
(AIAA  2005-6310).  Presented  at  the  AIAA  Modeling  and  Simulation  Technologies  Conference  and 
Exhibit,  San  Francisco,  California. 

4.3.3  Reasons  for  Migrating  Operator  Control 

4.3.3. 1  Limitations  Necessitating  Control  Migration 

4. 3. 3. 1.1  Temporal  Limits 

Many  current  human-machine  operations  are  continuous  in  character  and  the  nature  of  these  operations  often 
precludes  a  temporary  shutdown  because  of  economical  or  other  constraints  [1].  Such  operations  also  have  the 
potential  to  create  situations  in  which  people  are  driven  to  work  continuously.  This  has  certainly  been  the  case 
in  military  operations  where  technological  advances  in  night  vision  devices  and  other  sensors  coupled  with  a 
global  battle  space  has  led  to  a  doctrine  of  continuous,  around-the-clock  operations  [2,3].  Thus,  current 
military  endurance  UAV  systems  operate  at  distances  requiring  beyond-line-of-sight  communications  and  can 
remain  airborne  for  nearly  1-2  days.  Future  military  and  civil  UMV  systems  are  projected  to  operate  for 
durations  of  days  to  months  at  a  time  [4,5,6]. 

A  critical  problem  for  such  endurance  UMV  systems  is  the  predictable  decrements  experienced  by  individuals 
continuously  performing  cognitive  tasks  for  sustained  periods  [1,3].  In  a  study  comparing  the  effects  of 
fatigue  versus  alcohol  intoxication,  Dawson  and  Reid  [7]  found  the  hourly  performance  decrement  for  each 
hour  of  wakefulness  between  10  and  26  hours  was  equivalent  to  the  performance  decrement  observed  with  a 
0.004  percent  rise  in  blood  alcohol  concentration.  Therefore,  after  17  hours  of  sustained  wakefulness, 
cognitive  psychomotor  performance  decreases  to  a  level  equivalent  to  the  performance  impairment  observed 
at  a  blood  alcohol  concentration  of  0.05  percent.  This  is  the  proscribed  level  for  alcohol  intoxication  in  many 
western  industrialized  countries  [2,7]  and  exceeds  the  0.04  percent  limit  established  by  the  Federal  Aviation 
Administration  for  aircrew  [8].  Mullaney,  et  al.  [9,10,11]  conducted  studies  of  continuous  performance  on 
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monotonous  tasks  requiring  sustained  attention  and  found  such  performance  produced  rapid  fatigue  effects 
after  only  6  hours.  More  than  half  of  the  study  participants  experienced  psychological  disturbances  such  as 
mild  hallucinations,  illusions,  disorientation,  and  derealizations,  mostly  after  18  hours.  Finally,  the  operational 
impact  of  fatigue-related  performance  decrements  was  demonstrated  by  an  Air  Force  Safety  Center  study 
which  found  fatigue  was  present  in  12.7  percent  of  the  most  serious  class  of  Air  Force  mishaps  occurring 
during  fiscal  years  1972  -  2000  [12],  costing  the  Air  Force  an  estimated  54  million  dollars  each  year  [2]. 

While  it  is  obviously  unreasonable  to  expect  a  single  operator  to  control  a  UMV  with  an  endurance  greater 
than  1  day,  it  is  also  evident  operators  will  need  to  be  changed  during  UMV  operations  spanning  more  than 
the  majority  of  a  day.  Summarizing  NASA’s  experience  testing  UAVs,  Del  Frante  and  Cosentino  [13]  stressed 
the  importance  of  adhering  to  established  crew  rest  requirements,  implying  UMV  operations  are  not  immune 
to  fatigue-related  operator  performance  decrements.  Additionally,  studies  of  personnel  working  8,  10,  and 
12  hour  shifts  [14,15,16,17,18]  have  shown  increased  fatigue  and  poorer  performance  with  12-hour  versus 
eight  or  ten  hour  shifts.  Collectively,  these  studies  suggest  migration  of  operator  control  is  a  necessity  as 
UMV  endurances  routinely  exceed  12  hours,  although  optimally  it  should  be  considered  after  8-10  hours. 

4. 3. 3. 1.2  Physical  Limits 

As  previously  mentioned,  some  military  endurance  UAVs  currently  operate  at  great  distances  from  the  control 
station,  necessitating  beyond-line-of-sight  communications  [5,6].  As  satellites  or  other  UMVs  are  used  to 
relay  signals  over  the  horizon,  variable  time  delays  or  latencies  of  one  or  more  seconds  are  introduced. 
However,  latencies  greater  than  one  second  mean  real-time  feedback  necessary  for  effective  manual  control  is 
not  available  [19].  Additionally,  many  UMV  operators  are  dependent  on  real-time  imagery  from  cameras 
mounted  on  the  remote  vehicle  in  order  to  manually  control  the  vehicle  [20].  This  requires  data  transmission 
links  between  the  vehicle  and  control  station  with  high  bandwidths  and  low  latencies,  but  increasing  distance 
between  the  vehicle  and  control  station,  especially  beyond  line-of-sight,  necessarily  forces  constraints  on  data 
link  bandwidth  and  latency  [20,21,22].  Such  data  link  constraints  can  result  in  limited  temporal  resolution, 
spatial  resolution,  color,  and  field  of  view  of  imagery  irrespective  of  onboard  sensor  capabilities  [20,23]. 
With  great  enough  distances,  the  frequency  and  immediacy  of  transmitted  images  may  decrease  to  the  point 
where  direct  manual  control  of  the  vehicle  is  significantly  degraded  [21,24].  Besides  vehicle  control, 
experimental  evidence  has  also  shown  visual  tasks  such  as  target  detection  and  identification,  tracking, 
and  orientation  are  affected  by  degraded  image  quality,  slow  update  rates  (<  2  Hz)  and  high  latency  (>  2  sec) 
[20,25]. 

Increased  automation  (e.g.,  supervisory  control)  and  predictive  displays  have  been  utilized  to  mitigate  the 
effects  of  control  latency  [24,19].  However,  there  are  situations  where  manual  control  modes  may  be  preferred 
over  supervisory  control  or  a  fully  automated  vehicle  [20].  In  such  situations,  an  alternative  strategy  may  be  to 
handoff  control  to  more  proximate  control  stations.  This  approach  has  relative  merit  over  supervisory  control 
in  situations  where  concerns  over  control  latency  and  quality  of  visual  imagery  or  sensor  information  are 
critical,  such  as  in  highly  dynamic  and  changing  environments  where  near-real  time  data  is  required  for 
complex  decision-making  [21,20].  These  situations  might  include  weapons  employment  [24]  when  there  is  a 
risk  for  fratricide  or  responding  to  a  malfunctioning  or  damaged  remote  vehicle.  Additionally,  some  current 
military  UAVs  by  design  must  be  manually  controlled  during  takeoff  and  landing  [21,6,20]. 
In  these  circumstances,  a  control  station  needs  to  be  located  in  line-of-sight  distance  of  the  airfield  to 
minimize  data  transmission  delays  and  optimize  manual  control.  However,  once  airborne,  control  is  handed 
off  to  a  geographically  remote  control  station  where  the  UAV  is  operated  via  supervisory  control,  thereby 
minimizing  the  equipment  and  personnel  which  must  be  sent  to  a  potentially  vulnerable  forward  base  of 
operations. 
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4. 3. 3. 1.3  Functional  Limits 

Migration  of  control  within  a  crew  may  occur  when  a  UMV  system  is  designed  such  that  control  is 
functionally  divided  between  non-equivalent  controllers.  For  instance,  current  military  UAVs  are  typically 
operated  by  at  least  two  operators  with  one  responsible  for  vehicle  control  and  the  other  for  payload  control 
[21,20].  While  the  payload  operator  usually  does  not  directly  control  the  vehicle,  it  may  be  possible  for  this 
individual  to  exercise  indirect  control  of  a  UMV  if  it  is  designed  to  autonomously  adjust  its  path  to  stabilize 
camera  or  sensor  images.  Functional  migration  of  control  also  occurs  in  some  military  tactical  UAVs 
(TUAVs)  where  responsibility  for  takeoff/landing  and  en  route  control  is  divided  between  two  operators. 
In  this  situation,  the  external  pilot  (EP)  interacts  with  the  TUAV  while  in  direct  visual  contact  at  the  site  of 
takeoff  or  landing.  In  contrast,  the  air  vehicle  operator  (AVO)  is  inside  a  control  station  and  interacts  with  the 
TUAV  through  an  interface  of  sensor  displays  and  controls  during  the  en  route  phase  of  flight  [26,6,27]. 
While  control  is  functionally  divided  in  part  out  of  necessity  because  of  task-specific  human-machine 
interfaces  (HMIs),  Barnes  et  al.  [26]  also  demonstrated  differences  between  EPs  and  AVOs  with  regards  to 
operator  abilities  across  multiple  cognitive  skill  sets. 


4.3.3.2  Advantages  of  Control  Migration 

4. 3. 3. 2.1  Mitigate  Fatigue  and  Optimize  Operator  Vigilance 

In  highly  automated  systems  such  as  UMVs,  much  of  the  operator’s  task  load  is  supervisory  in  nature, 
consisting  mainly  of  passive  monitoring  of  system  parameters  and  remaining  alert  for  malfunctions  [28,20]. 
This  trend  towards  placing  the  operator  in  the  role  of  passive  monitor  has  continued  despite  years  of  vigilance 
research  demonstrating  such  roles  make  maintaining  a  constant  level  of  alertness  exceedingly  difficult 
[29,30,31,32]  and  predispose  to  “hazardous  states  of  awareness”  [33].  Studies  of  vigilance  tasks  have 
consistently  demonstrated  a  vigilance  decrement  beginning  as  early  as  20  -  35  minutes  after  initiation  of  a 
task  and  characterized  by  declining  numbers  of  correct  responses  and/or  increasing  response  times  [29,3,19]. 
One  study  found  declining  detection  rates  after  only  2-3  minutes  of  task  performance,  with  target  detection 
rates  eventually  plateauing  at  70  -  80  percent  of  initial  rates  [34].  Prolonged  vigilance  work  generally  invokes 
subjective  feelings  of  boredom  and  monotony  and  invariably  induces  decreased  levels  of  physiologic  arousal. 
Boredom  in  particular  can  become  apparent  within  minutes  of  the  onset  of  a  monotonous  task  and  is 
associated  with  decreased  performance  efficiency  and  increased  drowsiness.  However,  when  coupled  with  the 
need  to  maintain  high  levels  of  alertness,  vigilance  tasks  can  be  perceived  as  quite  stressful  [3,35].  This  stress 
predisposes  the  operator  to  short  term  fatigue  which  typically  manifests  as  long  response  times,  missed 
signals,  and  brief  interruptions  in  performance  due  to  gaps  or  lapses  in  attention  [36]  as  well  as  increased 
decision  errors  or  decreased  thruput  (e.g.,  maintain  accuracy  at  the  expense  of  performance  speed)  [3]. 
Thus,  it  should  be  expected  that  tasks  requiring  the  sustained  attention  of  UMV  operators  will  be  susceptible 
to  degraded  performance  and  increased  risk  for  operational  errors  [28]. 

Although  initial  research  [1,37]  with  complex  monitoring  tasks  typical  of  the  ATC  task  environment 
suggested  vigilance  decrements  did  not  occur,  more  recent  studies  are  supportive  of  the  vigilance  decrement 
in  both  simple  and  complex  monitoring  tasks  [29,39,36,40].  The  validity  of  these  concerns  in  UMV  operations 
was  demonstrated  in  a  study  of  Army  UAV  operator  performance  under  two  experimental  conditions 
involving  8-10  hour  versus  3  hour  flights  [41].  Target  detection  and  recognition  performance  as  well  as  crew 
reaction  times  were  significantly  degraded  during  nocturnal  operations  involving  the  longer  flights  while  no 
nocturnal  changes  were  observed  for  the  shorter  flights.  Likewise,  two  studies  [36,40]  using  an  ATC  task 
found  the  time  to  detect  and  the  frequency  of  missed  traffic  conflicts  increased  significantly  over  the  course  of 
just  2  hours.  These  vigilant  decrements  were  attributed  to  increasing  lapses  in  attention  rather  than  a 
generalized  fatigue  effect. 
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One  of  the  best  ways  to  overcome  these  effects  is  change,  whether  using  work  breaks,  rest  pauses,  or  split 
shifts,  although  the  benefits  of  rest  pauses  may  derive  more  from  subjective  factors  such  as  relief  of  boredom 
[3].  Warm  [42]  in  particular  recommended  continuous  vigilance  monitoring  tasks  be  kept  to  less  than  4  hours 
in  duration.  Regardless  of  the  method  of  change,  it  will  necessarily  involve  the  migration  of  control  to  another 
operator,  whether  in  the  same  or  another  control  station.  Thus,  migration  of  operator  control  plays  a 
potentially  critical  role  in  the  maintenance  of  optimum  operator  performance  and  decreasing  risk  for 
operational  errors. 

4.33.2.2  Operator  Functional  Specialization 

Unlike  manned  vehicles  where  crew  size  is  limited  by  vehicle  payload  constraints  and  all  operator  functional 
capabilities  must  be  resident  in  the  onboard  crew,  UMVs  offer  a  distinct  advantage  of  allowing  these 
functional  capabilities  to  be  distributed  over  a  multitude  of  potentially  geographically  dispersed  specialists. 
In  essence,  a  UMV  crew  is  limited  only  by  data  transmission  link  accessibility.  Given  the  massive  amounts  of 
information  currently  down-linked  from  UMV  systems  [28]  and  the  information  processing  constraints 
necessarily  imposed  by  the  sequential  decision-making  of  human  operators  [43],  it  is  key  to  distribute  this 
information  so  it  can  be  more  efficiently  processed  for  mission  accomplishment  [28].  This  point  was 
highlighted  by  a  study  of  human  systems  integration  (HSI)  issues  in  military  UAV  mishaps  which  found  an 
association  between  failures  in  the  cognitive  domain  and  operational  errors  [44]. 

Beyond  the  issue  of  information  processing,  UMVs  offer  the  opportunity  to  employ  task  specialization  beyond 
the  level  hereto  seen  in  traditional  manned  vehicles.  The  potential  need  for  task  specialization  in  UMV 
operations  can  be  understood  given  the  current  military  experience  with  UAVs  in  which  training  and 
attentional  issues  are  frequent  causal  factors  in  human  error-related  mishaps  [45].  As  noted  by  Barnes  et  al. 
[26]  in  their  study  of  Army  UAV  operators,  experience  improves  operators’  cognitive  throughput,  allowing 
them  to  devote  limited  attentional  resources  to  future  problems  while  automatically  attending  to  immediate 
perceptual  and  motor  tasks.  This  was  echoed  in  NASA’s  lessons  learned  testing  UAVs  [13]  that  “even  more 
important  than  practicing  the  emergency  procedures  is  practicing  the  normal  procedures  to  the  point  that  they 
are  second  nature”  so  anomalies  are  addressed  with  increased  attention.  Thus,  experience  serves  to  increase  an 
operator’s  cognitive  efficiency  in  problem  solving  by  effectively  increasing  attentional  resources  [19]. 
Unfortunately,  the  cognitive  efficiency  obtained  via  experience  is  specific  to  the  tasks  experienced  and  not 
broadly  generalizable  [26]. 

Given  the  task-centric  nature  of  expertise,  consideration  should  be  given  to  the  creation  of  specialty  teams  to 
intervene  and  handle  uncommon  or  off-nominal  events.  Such  teams  could  rehearse  infrequent  tasks  and 
explore  potential  outcomes,  thereby  developing  proficiency  in  situational  problem  solving  prior  to 
encountering  the  actual  event.  For  example,  rather  than  training  all  operators  to  handle  emergencies,  specialty 
teams  could  be  trained  to  take  control  of  the  vehicle  and  troubleshoot  a  malfunctioning  or  damaged  UMV. 
These  emergency  teams  could  operate  from  remote  control  stations  equipped  with  enhanced  displays  to  aid 
diagnosis  and  allow  predictive  simulation.  Such  methods  have  been  used  successfully  in  the  U.S.  space 
program  [28]. 

Functional  specialization  may  also  be  utilized  in  non-emergent,  nominal  situations  to  optimize  the  central  role 
of  the  human  decision-maker  within  the  system.  In  traditional  manned  combat  vehicles,  the  local  operator  has 
the  responsibility  to  authenticate  targets  and  trigger  weapons  because  they  are  presumed  to  be  in  the  best 
position  to  assess  the  situation  and  make  timely  decisions.  However,  UMVs  allow  this  function  to  be  migrated 
to  other  team  members  possessing  higher  levels  of  technical  or  combat  expertise  such  as  a  target  detection 
specialist  or  experienced  mission  commander,  thereby  improving  the  confidence  level  of  information 
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presented  to  controlling  authorities  [28].  Additionally,  functional  specialization  allows  for  increased  training 
program  efficiency  since  all  personnel  don’t  need  to  receive  equal  training.  The  FAA  has  explored  this  issue 
with  regards  to  training  air  traffic  control  specialists  [46]  and  the  Air  Force  has  adopted  this  approach  in 
training  MQ-1  Predator  operators  on  takeoff  and  landing. 

4.33.2.3  Workload  and  Multiple  Vehicle  Control 

There  currently  is  a  limited  body  of  human  factors  research  suggesting  one  operator  may  control  more  than 
one  UMV  under  relatively  idealized  conditions  to  include  closely  coordinated  and  correlated  vehicle 
activities,  a  stable  environment,  and  reliable  automation  [38,47,48,6].  However,  research  has  also 
demonstrated  that  operator  performance  controlling  a  single  vehicle  is  significantly  degraded  when  heavy 
demands  are  imposed  by  payload  operations  [41,49,25].  This  would  suggest  the  ability  of  an  operator  to  attend 
to  multiple  UMVs  may  be  severely  compromised  under  non-idealized  conditions,  especially  if  one  vehicle  is 
malfunctioning  or  damaged  [6].  Additional  human  factors  research  is  available  examining  situational 
awareness  in  ATC,  an  analog  for  supervisory  control  of  multiple  vehicles.  Endsley  and  Rodgers  [50]  found  air 
traffic  controller  accuracy  was  significantly  impacted  by  the  number  of  aircraft  being  controlled  and 
situational  awareness  declined  dramatically  when  the  number  of  aircraft  exceeded  8-10.  This  is  consistent 
with  the  magic  number  “7  ±  2”  in  memory  research  [51]  which  states  there  are  finite  limits  on  human 
information  processing  beyond  which  people  tend  to  forget.  Two  studies  [52,22]  examining  air  traffic 
controller  performance  when  passively  monitoring  aircraft  under  free  flight  conditions  found  a  significant 
decrement  in  situational  awareness  when  controllers  had  to  handle  17  versus  11  aircraft.  Additionally,  a  study 
examining  control  of  multiple  retargetable  missiles  [38]  found  operators  could  effectively  control  8  - 
12  missiles,  but  performance  degraded  with  16  missiles.  The  preponderance  of  the  evidence  thus  suggests 
greater  than  12  vehicles  potentially  represents  a  cognitive  saturation  state  for  controllers  interacting  with  semi- 
autonomous  vehicles  requiring  only  the  setting  of  new  goals  [53]. 

Given  the  suggested  limits  to  an  operator’s  ability  to  manage  multiple  UMVs,  migration  of  control  may  be 
utilized  as  a  workload  mitigation  strategy.  For  example,  an  operator  controlling  multiple  UMVs  under  high 
workload  conditions  (e.g.,  multiple  vehicle  requests  for  operator  attention)  could  transfer  control  of  one  or 
more  UMVs  to  other  operators  under  low  workload  conditions,  even  as  part  of  normal  operations.  In  cases 
where  a  single  UMV  requires  sustained  attention  because  of  a  backlog  of  vehicle  requests  for  operator 
attention  or  off-nominal  operating  conditions,  control  of  this  high  workload  UMV  could  be  transferred  to  an 
on-call  operator  or  supervisor  akin  to  current  ATC  practices.  The  ability  to  transfer  control  of  UMVs  based  on 
workload  prevents  a  single  operator  controlling  multiple  UMVs  from  having  to  adopt  triage-like  procedures 
for  handling  multiple  attentional  demands  [28]. 

4.33.2.4  Payload  Control 

As  already  alluded  to  during  the  discussion  of  multiple  vehicle  control,  performing  payload  tasks  can 
significantly  increase  operator  workload  and  degrade  operator  performance.  In  current  military  reconnaissance 
UAVs,  vehicle  and  payload  control  are  typically  divided  between  a  vehicle  operator  and  a  payload  operator 
[21,25].  Van  Erp  and  Van  Breda  [25]  concluded  such  a  crew  structure  was  reasonable  in  light  of  findings  that 
consolidating  vehicle  and  payload  control  within  a  single  operator  substantially  degraded  performance. 
Likewise,  Barnes  and  Matz  [41]  studied  a  UAV  control  station  configuration  which  required  a  single  operator 
to  perform  both  aviation  and  target  acquisition  functions.  They  found  operators  became  focused  on  the 
targeting  function  to  the  detriment  of  situational  awareness  and  vehicle  control,  leading  the  authors  to  question 
the  wisdom  of  using  a  single  operator.  Two  additional  studies  demonstrated  attentional  fixation  and  cognitive 
tunneling  during  target  analysis  which  degraded  performance  on  other  tasks  irrespective  of  level  of 
automation  or  use  of  auditory  cueing  [47,49].  The  task  of  manipulating  a  camera  image,  analyzing  targets, 
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and  keeping  track  of  cardinal  directions  appears  to  be  sufficiently  challenging  that  timesharing  with  other 
tasks  such  as  vehicle  control  becomes  virtually  impossible.  Finally,  Van  Breda  and  Passenier  [54]  noted 
operator  performance  was  poor  when  utilizing  “double-stick”  controls  where  one  joystick  controls  airframe 
heading  and  speed  and  the  other  camera  heading  and  pitch.  However,  this  is  not  surprising  given  such  control 
arrangements  predispose  to  perceptual  confusion  which  increases  the  potential  for  action  slip  errors  [19]. 

Nevertheless,  for  optimal  flexibility  and  cost  effectiveness,  it  is  desirable  to  allocate  both  vehicle  and  payload 
control  to  one  operator  [54].  It  may  therefore  be  necessary  to  delineate  circumstances  under  which  vehicle  and 
payload  control  may  be  safely  performed  by  a  single  UMV  operator  and  when  control  of  the  payload  should 
be  transferred  to  another  operator  [6].  This  may  be  decided  prior  to  a  mission  or  payload  control  may  need  to 
be  transferred  during  a  mission  in  response  to  a  dynamically  evolving  situation.  As  previously  discussed, 
a  UMV  operator’s  ability  to  perform  payload-oriented  visual  tasks  such  as  target  detection  and  identification, 
tracking,  and  orientation  is  impaired  by  low  temporal  update  rates  and  long  transmission  delays  [20,25]. 
If  vehicle  control  is  adequate  for  the  task  using  some  form  of  supervisory  control,  it  may  only  be  necessary  to 
handoff  payload  control  to  a  more  proximate  control  station,  potentially  eliminating  the  need  for  full  control 
stations  in  forward  locations.  Additionally,  the  ability  to  handoff  payload  control  to  those  directly  requesting 
the  camera  imagery  or  sensor  data  (e.g.,  target  detection  specialists  or  fielded  forces)  could  increase  the 
efficiency  of  data  collection  and  eliminate  the  need  for  coordination  with  an  intermediate  payload  operator. 
At  the  extreme,  control  of  weapons  could  be  transferred  to  the  forces  requesting  their  employment,  hopefully 
decreasing  the  risk  for  fratricide. 

4.3.3.3  Disadvantages  of  Migrating  Operator  Control 

4. 3. 3. 3. 1  Situational  Awareness 

While  there  are  good  reasons  to  consider  utilizing  migration  of  operator  control  in  UMVs,  it  is  important  to 
also  explore  the  potential  pitfalls.  Indeed,  migration  of  control  may  well  constitute  a  critical  and  potentially 
high  workload  phase  for  UMV  operators  [6].  For  example,  several  military  UAV  accidents  have  occurred 
either  directly  during  or  indirectly  as  the  result  of  changeovers  or  handoffs  [6,55,27].  In  handing  off  control 
between  stations,  mishaps  have  resulted  when  the  station  receiving  control  was  improperly  configured  [27]. 
During  changeovers,  mishaps  have  resulted  because  of  the  new  operator’s  decreased  systems  awareness  [55]. 
More  broadly,  there  is  also  concern  for  an  acute  decrement  in  crew  situational  awareness  when  control  is 
transferred  to  a  crew  not  currently  involved  in  the  ongoing  mission  [55].  Kidd  and  Kinkade  [1]  demonstrated 
the  existence  of  such  an  operator  change-over  performance  decrement  in  the  ATC  environment.  Controller 
performance  was  markedly  decreased  over  the  first  5  minute  period  following  assumption  of  controller  duties. 
This  change-over  performance  decrement  was  mitigated  by  about  50  percent  if  either  parallel  control  or 
auditory-plus-visual  monitoring  was  employed  as  a  pretransition  condition.  Another  study  examining 
operational  errors  in  ATC  [56]  found  errors  were  most  frequent  during  the  first  15  minutes  after  assuming 
controller  duties  and  nearly  half  occurred  within  the  first  30  minutes  on  position.  Likewise,  a  study  of  Army 
UAV  operators  [41]  found  operators  preferred  longer  over  shorter  rotations  because  they  perceived  the  longer 
rotations  allowed  for  better  situational  awareness  of  the  tactical  environment. 

4. 3. 3. 3. 2  Complex  Teaming  Situations 

Migrating  operator  control  can  create  distributed  crews  with  dynamically  changing  membership  which  may 
have  significant  associated  costs  in  terms  of  the  increased  complexity  of  crew  coordination  and 
communication  [57,6].  Breakdowns  in  team  performance,  cooperation,  and  communication  have  been  shown 
to  be  a  contributing  factor  in  military  UAV  mishaps  [45].  Complex  teaming  situations  dictate  the  need  for 
highly  efficient,  structured,  and  reliable  communication.  Unfortunately,  the  nature  of  human-human  and 
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human-machine  teams  in  high  stress  environments  tends  to  create  an  unfavorable  climate  for  successful 
communication.  With  respect  to  human  behavior,  emotion,  information  overload,  and  a  lack  of  situational 
awareness  comprise  a  few  of  the  factors  which  impede  the  production  and  processing  of  messages,  resulting  in 
low  fidelity  communication. 

Supporting  complex  teaming  situations  is  further  complicated  by  the  densely  bureaucratic  nature  of  military 
and  governmental  bodies.  The  culture  and  structure  of  these  types  of  organizations  has  historically  been 
characterized  by  rigid,  formal,  and  delayed  communication.  Large,  structured  networks  constrain 
communication,  which  results  in  delayed  information  processing  and  a  lagging  response  to  environmental 
stimuli.  A  change  in  organizational  structure  from  a  traditional  management  hierarchy  to  a  self-organizing 
system  of  self-managed,  mixed-entity  teams  can  improve  the  military’s  ability  to  respond  to  critical  C2 
situations. 
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4.3.4  Effects  of  Control  Migration  on  Concept  of  Teams 

4.3.4.1  Teams  Versus  Groups 

A  group  refers  to  a  small  set  of  organizational  members  comprising  three  to  fifteen  actors  who  regularly 
interact  for  the  purpose  of  a  common  goal.  All  teams  qualify  as  groups;  however,  all  groups  do  not  fulfill  the 
criteria  required  for  classification  as  a  team.  Teams  are  behaviorally  distinct  from  groups  on  the  dimensions  of 
performance  requirements,  interdependence,  and  accountability.  “A  set  of  two  or  more  individuals  who  are 
expected  and  required  to  interact  dynamically,  interdependently,  and  adaptively  to  accomplish  their  goals, 
but  do  not  are  still  a  team  -  they  are  simply  an  ineffective  team”  [1].  The  existence  and  prominence  of 
teaming  in  organizations  is  a  significant  distinguisher  of  organizational  structure  and  culture. 

4.3.4.2  Types  of  Teams 

Teams  may  be  classified  according  to  four  general  structures:  traditional  work  teams,  long-term  project  teams, 
network  design  structures,  and  parallel  teams  [2].  These  structures  may  be  further  characterized  by  formality, 
temporality,  purpose,  and  membership.  Therefore,  teams  may  be  distinguished  from  one  another  in  terms  of 
formal  designation  within  a  system,  lifespan  of  the  team,  tasks  and  functions,  designation  of  goals, 
interchangeability  of  skills,  and  the  extent  to  which  team  members  possess  membership  on  multiple  teams. 
An  additional  aspect  is  the  notion  of  distribution;  do  team  members  reside  locally  or  remotely  within  and/or 
external  to  the  organization?  This  is  a  critical  factor  because  a  distributed  or  virtual  team  demands  highly 
flexible,  coordinated,  and  specific  communication.  Watson-Manheim  and  Belanger  [3]  noted  the  dependence 
of  virtual  teams  on  communication  technologies  for  accomplishing  goals,  yet  these  same  teams  experienced 
significant  problems  with  communication-based  work  processes. 

As  the  relationship  of  human  and  computer  interaction  evolves,  the  notion  of  what  has  constituted  a  human- 
member  team  must  be  changed  to  include  nonhuman  entities  (e.g.,  autonomous  UMVs).  This  broadening  of 
relationship  definition  can  still  be  loosely  characterized  within  the  traditional  definition  of  team. 
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The  migration  of  operator  control  to  autonomous  agents  is  grounded  in  the  need  of  team  entities  to  achieve 
mission  success.  Because  teams  have  greater  performance  than  work  groups,  this  fulfills  the  criteria  of 
meeting  performance  requirements.  Consequently,  optimal  mission  performance  with  respect  to  migration  of 
operator  control  is  dictated  by  the  interdependence  of  humans  and  machines.  Successful  decision  making  is 
determined  by  the  cooperation  and  coordination  of  all  team  entities.  The  third  criterion  is  illustrated  by  the 
shared  accountability  of  man  and  machine  in  mission  success.  Have  the  appropriate  exchanges  occurred  for 
operator  control  from  human  to  machine  and  vice  versa? 

Successful  team  function  is  dependent  upon  team  member  collaboration,  trust,  and  technology.  One  aspect  of 
how  communication  is  important  is  its  role  in  information  behavior.  Sonnenwald  and  Pierce  [4]  determined 
the  why,  what,  and  how  were  relevant  concepts  for  group  work  contexts  in  C2  environments.  “C2  team 
members  need  to  collect,  synthesize  and  disseminate  information  to  create  an  understanding  of  the  current 
battlefield  situation  and  to  anticipate  future  battlefield  events”.  One  should  also  consider  the  who  and  when. 
Therefore,  military  teams  can  be  understood  as  functional  organisms,  coordinating  work  processes  among 
team  actors  to  achieve  macro  and  micro  level  system  goals  within  specified  timeframes. 

Coalition  operations  require  the  coordination  of  dynamic  and  culturally-diverse  teams.  Regardless  of  whether 
the  teams  comprise  humans,  UMV  members,  or  both,  the  principles  of  intercultural  communication  may 
apply.  Cross-cultural  differences  may  present  challenges  for  facilitating  intra-team  and  inter-team 
coordination.  These  differences  can  be  evident  in  team  member  attitudes  and  behaviors,  and  more  recently, 
have  been  attributed  to  primary  differences  in  group  cognition  and  behavior  [5].  Collectively,  these  variables 
can  cause  undesirable  or  hazardous  behavior  during  team  operations.  Team  interactions  should  be  structured 
to  manage  uncertainty  and  anxiety  for  face-to-face  and  virtual  contexts. 

4.3.4.3  Distributed  and  Dynamically  Changing  Teams 

Virtual  teams  are  groups  in  which  the  members  are  geographically  distributed  but  technologically  connected 
to  support  asynchronous  and/or  synchronous  communication  for  the  purpose  of  coordinating  behavior  and 
goal  achievement.  In  this  context,  technology  platforms  are  critical  foundations  for  forging  network  relations 
and  permitting  teamwork.  Virtual  teams  provide  an  organizational  structure  contrary  to  the  high-level 
hierarchies  of  military  and  government  bureaucracies.  The  result  reflects  a  flattened,  lean,  fluid,  specialized 
and  responsive  network. 

Qureshi  and  Vogel  [6]  identified  three  types  of  adaptive  social  processes  virtual  teams  experience:  technological, 
work,  and  social  adaptation.  The  adoption  of  new  technologies  for  communication,  the  application  and 
integration  of  technologies  to  team  member  ways  of  working,  and  the  emergence  of  social  norms  related  to 
technology  affect  the  way  virtual  teams  adapt  to  the  environment.  Virtual  teams  are  communication  and 
technology  intensive.  “The  challenge  for  virtual  teams  is  in  allocating  tasks  based  on  knowledge  and  skill  in 
an  environment  that  is  often  dispersed  across  space,  time  and  organizations”.  This  environment  is  further 
complicated  by  the  communication  demands  of  culturally-diverse  team  members.  Lee-Kelley,  Crossman, 
and  Cannings  [7]  suggest  a  “focus  on  attributes  of  persons,  situational  expressions  of  these  attributes  and  the 
relation  of  personal  attributes  toward  others  in  the  work  group,  may  be  used  to  explain  some  of  the  relational 
issues  that  spatial  and  temporal  separation  in  the  virtual  team  may  create”.  Managers  and  organizational 
designers  should  consider  the  additional  complexity  of  cultural  diversity  and  how  it  impacts  relational  issues 
among  team  members. 

In  developing  communication  protocols  for  UMV  teams,  the  dimensions  of  virtual  organizations  proposed  by 
DeSanctis,  Staudenmayer,  and  Wong  [8]  may  be  useful.  The  characterization  of  space  (e.g.,  spatial  dispersion 
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of  organizational  members),  time  (e.g.,  the  degree  of  asynchronous  operation),  culture  (e.g.,  norm,  value, 
behavioral  variability),  and  boundary  (e.g.,  organizational  dispersion)  provides  a  framework  for  developing 
communication  standards.  As  previously  noted,  virtual  teams  have  similarities  when  compared  with 
traditional  team  structures;  however,  because  of  the  heightened  dependency  on  information  communication 
technology  (ICT),  understanding  the  organizational  processes  at  the  supra  and  micro  system  levels  becomes 
more  critical  for  virtual  teams.  Only  when  this  knowledge  is  obtained  and  observed  can  virtual  teams  be 
effectively  and  properly  supported.  For  example,  the  transfer  of  UMV  system  control  from  operator  to 
operator  is  a  team  process.  By  understanding  the  vigilance  issues,  the  constraints  of  operator  control  transfers, 
and  other  environmental  variables,  ICT  can  be  optimally  structured. 


4.3.4A  Dimensions  of  Team  Performance 

4. 3. 4. 4. 1  Organizational  Culture  and  Structure 

Organizational  processes  occur  at  every  system  level:  the  individual  components,  the  environmental  context, 
the  system  goals,  and  the  organizational  resources  (e.g.,  people,  technology,  etc.).  When  a  keen  understanding 
of  the  inner  workings  of  a  system  is  established,  leaders  occupy  an  educated  position  to  plan  and  manage 
change.  Ali,  Pascoe,  and  Warne  [9]  reinforced  findings  from  the  literature  regarding  the  role  of  organizational 
culture  in  effective  social  learning  and  organizational  change.  Organizations  which  sustain  an  environment  of 
member  empowerment,  forgiveness,  trust,  individual  and  organizational  commitment,  information  sharing, 
openness  of  decision-making,  and  cultural  cohesiveness  enable  effective  social  learning.  Organizations  which 
uphold  these  values  create  environments  where  members  are  able  to  engage  in  quality  learning.  Open 
communication  is  a  primary  catalyst  for  achieving  organizational  learning  [10].  “Social  control  is  particularly 
valuable  when  the  need  for  sharing  tacit  knowledge  increases  over  socially  impoverished  channels  of  virtual 
communication,  where  conflict  may  escalate  due  to  teamwork  issues  engendered  by  cultural  difference  in 
communication  and  problem-solving  styles  and  approaches”  [11]. 

Organizations  must  maintain  high  levels  of  adaptability  to  achieve  an  optimal  level  of  survival/success. 
“An  adaptive  structure  requires  organizations  to  develop  dynamic  capabilities  to  modify  current  practices  in 
response  to  dynamic  changes  in  the  environment”  [11].  Many  have  applied  a  deterministic  approach  to 
communication  technology,  and  view  technology  as  the  source  of  impact  in  a  social  network.  A  social 
network  approach  indicates  the  converse.  According  to  a  social  construction  perspective,  the  relationship  of 
technology  and  structure  is  indicated  by  a  simultaneous  affecting  of  each  other  [12];  “the  composition  of 
content  and  context  are  not  predetermined  by  technological  design  or  by  the  prior  existence  of  certain  social 
groups”. 

Cultural  differences  may  exist  across  team  members  on  the  dimensions  of  learning  style,  thinking  style, 
teamwork  experience,  functional  expertise,  uncertainty  threshold,  and  intelligence  (e.g.,  analytical,  practical, 
and  creative)  [11].  These  differences  may  create  barriers  to  building  trust  and  consequently  retard  the 
development  of  team  cohesion  [7].  To  further  complicate  team  functioning,  language  usage  can  result  in 
inaccurate,  ambiguous  communication. 

High  performing  virtual  teams  tend  to  have  members  with  experience  in  multicultural  environments. 
“The  difficulty  of  getting  a  group  of  disparate  people  to  come  together  and  work  effectively  cannot  be  over 
emphasized”  [7].  One  approach  to  managing  intercultural  communication  is  to  minimize  adverse  behaviors  by 
building  trust  among  culturally-diverse  team  members  and  across  teams.  “The  status  report  genre,  bug/error 
notification  genre,  update  notification  genre,  and  phone  meeting  management  genre  system  emerged  as  key 
communication  structure  that  both  reflected  and  shaped  members’  temporal  and  work  practices”  [13]. 
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This  reflects  the  suitability  of  structuring  communication  and  work  processes  for  improved  coordination  in 
distributed  teams.  The  enactment  of  work  routines  provides  stability  for  team  structures.  Team  stability  is 
positively  affected  by  homogeneous  rules.  Heterogeneous  rules  are  associated  with  increased  novelty  and 
team  adaptation  [14].  Therefore,  it  is  advisable  to  conceive  of  the  appropriate  levels  of  stability  for  C2  teams 
and  balance  heterogeneous  and  homogeneous  behavioral  rules  accordingly. 

4. 3. 4. 4. 2  Satisfaction 

The  team  is  important  to  an  organization  because  its  effectiveness  determines  organizational  mission  success 
and  member  satisfaction.  The  literature  has  numerous  examples  of  research  supporting  the  relationship  of 
member  satisfaction  and  member  productivity.  With  the  transition  to  increasingly  autonomous  UMVs,  some 
might  dismiss  team  member  satisfaction  as  irrelevant.  However,  as  long  as  HMI  is  present,  satisfaction 
remains  important  for  the  human  elements  of  the  team.  High  satisfaction  of  team  members  varies  directly  with 
team  commitment  and  committed  team  members  are  critical  for  operational  success. 

4. 3. 4. 4. 3  Qualitative  Composition  of  Teams 

Teams  are  comprised  of  organizational  members;  each  member  presenting  emotional,  psychological, 
and  behavioral  differences.  One  dimension  of  personality  is  the  typing  of  individuals  as  A  or  B  (TABP). 
Keinan  and  Koren  [15]  established  a  significant  relationship  between  team  member  personalities  and  group 
performance.  “Our  findings  show  that  when  the  task  is  competitive,  the  effect  of  TABP  on  performance 
differs  from  when  the  task  is  noncompetitive.”  Teams  primarily  comprising  type  A  personalities  are  directly 
associated  with  improved  team  performance  for  competitive  tasks  and  are  generally  more  productive  than 
those  teams  with  a  majority  of  type  B  personalities. 

But,  the  literature  is  generally  contradictory  in  the  area  of  group  composition.  For  example,  homogeneous 
teams  have  been  reported  as  both  positive  and  negative  on  the  measure  of  performance.  It  has  been  argued 
trait  similarity  increases  team  member  attraction;  this  positively  affects  relational  satisfaction  and 
team  performance.  Conversely,  research  has  also  documented  the  positive  relationship  of  dissimilar 
group  membership  and  team  performance.  Theoretically,  members  with  varied  opinions,  knowledge, 
and  experiences  will  provide  a  greater  quantity  of  unique  problem  resolutions  resulting  in  optimal 
performance  for  complex  environments. 

Teams  comprising  fundamental  knowledge,  skills,  and  abilities  (KSAs)  are  better  equipped  to  fulfill  mission 
goals.  KSA  requirements  are  not  completely  transferable  from  in-person  teams  to  virtual  teams  and  vice  versa. 
The  densely  computer-mediated  communication  environment  of  the  virtual  realm  requires  a  heightened 
adeptness  at  managing  digital  conflict,  text-intensive  interactions,  and  media  selection  [16]. 

4. 3. 4. 4. 4  Trust 

Virtual  teams  are  not  immune  to  the  requirements  for  success  that  traditional  teams  face.  Team  effectiveness 
is  determined  by  connecting  with  key  team  members  for  information  sources  (e.g.,  knowing  who  to  call  for 
specific  questions),  using  appropriate  communication  media  (e.g.,  e-mail,  telephone,  listservs,  intranets,  etc.), 
and  establishing  a  positive  organizational  culture.  Watson-Manheim  and  Berlanger  [3]  found  virtual  teams  are 
highly  influenced  by  establishing  organizational  norms  for  communication  media  choice,  providing 
appropriate  training,  and  managing  team  member  relationships. 
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4. 3. 4. 4. 5  Training 

The  literature  indicates  teamwork  does  enhance  performance;  therefore,  teams  can  benefit  from  focusing  on 
improving  member  relations.  Training  is  a  primary  vehicle  for  creating  a  positive  teamwork  environment. 
This  may  include  building  skills  in  coordination  [17].  Team  members,  human  or  machine,  must  acknowledge 
communication,  provide  meaningful  information  at  appropriate  time  intervals,  use  standardized  language,  use 
feedback  mechanisms,  address  conflicts,  recognize  uncertain  and  complex  environments,  and  identify 
problems.  Teams  will  likely  benefit  from  training  in  conflict  management,  uncertainty  avoidance,  learning  to 
select  appropriate  communication  channels,  and  how  to  design  appropriate  messages.  Indeed,  training  is  an 
important  and  viable  component  of  performance  enhancement  programs.  Stout,  Salas,  and  Fowlkes  [1] 
suggest  “that  team  training  equates  to  providing  trainees  with  necessary  KSAs  (e.g.,  team  competencies) 
to  engage  in  cooperative  behavior  and  to  efficiently  interact  with  one  another  to  attain  effectiveness”.  In  their 
study  of  training  effects  in  complex  environments,  pilots  who  were  trained  in  teamwork  behaviors  were  better 
prepared  to  deal  with  complex  problems  using  team  competencies. 
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4.3.5  Important  Issues  in  Control  Migration 

4.3.5.1  Interoperability 

Implicit  in  the  concept  of  migration  of  operator  control  is  the  assumption  that  there  is  complete 
interoperability  of  both  systems  and  personnel.  Current  work  on  the  NATO  UAV  interoperability  effort 
(e.g.,  STANAG  4586)  aims  to  address  issues  of  system  interoperability  specifically  with  regards  to  the  data 
link  interface  between  the  control  station  and  vehicle  and  C2  interfaces  between  the  control  station  and 
external  command  and  control,  communications,  computers,  and  intelligence  (C41)  systems.  However, 
despite  these  efforts,  migration  of  operator  control  is  currently  regarded  as  one  of  the  most  complex  and  risky 
phases  of  UMV  operations.  For  example,  many  system  parameters  may  be  changed  and  difficult  procedural 
and  technical  issues  can  be  involved.  This  phase  typically  requires  meticulous  planning  and  allows  almost  no 
flexibility  in  execution.  Migrating  control  between  dissimilar  systems  is  particularly  difficult  because  of  issues 
of  system  synchronisation.  Migration  of  control  between  operators  and  systems  at  physically  dispersed 
locations  may  require  initiation  and  alignment  of  systems,  one  or  more  data  and  communications  links,  and 
possibly  even  cryptological  equipment.  It  may  also  require  coordination  with  external  C2  agencies. 
This  situation  may  be  made  more  complex  if  a  face-to-face  debrief  is  not  possible.  Additionally,  the  control 
system  will  need  to  be  designed  to  allow  for  system  synchronisation  and  facilitate  operators  achieving  an 
adequate  level  of  situational  and  system’s  awareness  so  a  handover  can  be  safely  performed.  However, 
standards  and  safeguards  are  particularly  lacking  for  the  latter  issue. 

4.3.5.2  Procedures  for  Migration  of  Control 

Using  the  example  of  current  UAV  systems,  migration  of  operator  control  needs  to  be  coordinated  prior  to  the 
actual  event.  This  means  the  specific  procedures  and  information  to  be  exchanged  should  be  identified  during 
the  mission  planning  process.  The  procedures  should  be  available  in  checklist  form  and  should  have  been 
previously  validated  to  minimize  the  unintended  effects  of  operator  input  errors  as  well  as  be  applicable  to 
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both  nominal  and  off-nominal  situations.  Information  exchange  can  be  facilitated  by  the  preparation  of  a 
mission  folder  containing  the  flight  plan,  tasking,  handover  location,  datalink  parameters  (e.g.,  frequencies 
and  cryptological  settings),  other  system  settings,  and  emergency  or  contingency  plans.  Additionally,  there 
needs  to  be  processes  implemented  to  ensure  this  information  is  updated  to  reflect  actual  mission 
circumstances.  Since  migration  of  operator  control  demands  a  high  level  of  crew  coordination,  all  involved 
personnel  should  have  initial  and  recurrent  proficiency  training  in  control  transfer  procedures  as  well  as  crew 
coordination.  The  latter  may  be  particularly  applicable  for  personnel  without  prior  aviation  experience. 
Finally,  systems  should  be  designed  to  provide  immediate  and  unambiguous  feedback  to  operators  regarding 
the  state  of  control  transfer,  whether  gaining  or  releasing. 


4.3.5.3  Team  Situational  Awareness 

4. 3. 5. 3. 1  Levels  of  Situational  Awareness 

Team  situational  awareness  has  been  described  as  the  process  by  which  a  knowledge-heterogeneous  team 
develops  a  dynamic  shared  mental  model  in  accordance  with  the  demands  of  making  predictions  about  a 
dynamic  team  task  environment  [1].  Recent  research  has  demonstrated  in  turn  that  team  performance  directly 
correlates  with  team  members’  levels  of  SA  [2].  Accordingly,  in  order  to  safely  migrate  operator  control, 
it  is  imperative  the  operator  gaining  control  have  at  least  the  same  level  of  SA  as  the  operator  releasing  control 
[3].  Endsley  [4]  has  described  three  levels  of  SA:  perception  of  the  elements  in  the  environment  (level  1  SA), 
comprehension  of  the  current  situation  (level  2  SA),  and  prediction  of  the  future  status  of  one’s  own  situation 
and  the  surrounding  elements  (level  3  SA).  Operators  should  strive  for  the  highest  level  of  SA  (e.g.,  level  3 
SA)  prior  to  assuming  control  of  a  UMV  [5].  SA  may  need  to  be  achieved  at  the  system,  operational, 
and  mission  levels.  For  a  UAV  mission,  this  may  include  SA  of  a  very  broad  array  of  issues  to  include: 

•  System  status:  Fuel  status,  power  settings,  active  and  non-active  systems,  payload  status,  settings,  etc. 

•  System  degradations:  Missing  functionality  and  its  consequences  for  flight  continuation,  vehicle 
performance,  achievement  of  mission  objectives,  etc. 

•  Datalink  status:  Coverage,  frequencies,  cryptological  settings,  antenna  alignment,  etc. 

•  Vehicle  parameters:  Position,  speed,  attitude,  intentions,  and  future  flight  path. 

•  Airspace:  Restricted  areas,  danger  zones,  and  both  current  and  predicted  weather. 

•  Position  of  other  elements  in  the  environment:  Traffic,  threats,  cooperative  elements,  and  coalition 
assets  as  well  as  their  intentions  and  predicted  future  status. 

•  Mission  objectives:  Tasking(s),  commander’s  intent,  target  information,  downstream  user 
requirements,  intelligence  situation  and  prognoses,  etc. 

Obviously,  level  1  SA  is  easier  to  obtain  and  requires  less  effort  than  level  3  SA.  However,  the  more  complex 
the  situation,  the  more  important  it  becomes  to  achieve  level  3  SA,  and  thus  the  more  complicated  the  process 
for  migration  of  operator  control. 

4. 3. 5. 3. 2  Sharing  Situational  Awareness 

Current  research  has  yielded  conflicting  results  on  the  existence  of  detrimental  effects  of  geographic 
dispersion  on  team  processes,  team  knowledge,  and  team  situational  awareness.  However,  co-located  teams 
appear  to  more  readily  carry  out  planning  and  adaptive  behaviours  and  generally  communicated  more  than 
distributed  teams  [6].  Consequently,  it  appears  important  to  facilitate  information  sharing  during  UMV 
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handoffs,  which  may  be  accomplished  in  several  ways.  It  can  involve  voice  communication,  text  message 
exchange,  graphical  display  exchange  or  alignment  of  system  information  (including  graphics  and  status). 
The  human  interaction  in  these  exchanges  can  vary  greatly.  However,  this  is  also  true  of  the  potential  for 
human  error  during  the  exchange.  The  more  complex  the  situation,  the  more  operators  will  have  to  rely  on 
automation  to  help  them  attain  SA  considering  control  transfers  have  to  be  performed  within  a  certain 
timeframe.  Therefore,  there  is  a  need  to  design  the  HMI  for  the  operator  in  such  a  way  to  minimize  the 
workload  during  transfer  of  control.  As  an  example  it  could  help  the  operator  to  focus  the  attention  to  changed 
mission  folder  items  compared  to  the  original  situation. 

4.3.5.4  Priorities  in  Sharing  Information 

Teams  need  environments  which  facilitate  efficient  and  effective  C2  information  sharing.  Communication 
principles  for  data  and  knowledge  exchanges  can  be  applied  at  the  human  and  human-computer  levels  in  C2 
contexts.  Information  priorities  may  be  classified  in  a  communication  taxonomy  for  UMV  operator  teams. 
When  team  members  trust  each  other  and  the  team  infrastructure,  are  educated  about  organizational  structure 
and  processes,  and  understand  information  processing,  fluid  communication  is  enabled. 

Teams  will  likely  benefit  from  training  in  information  salience,  conflict  management,  uncertainty  avoidance, 
learning  to  select  appropriate  communication  channels,  how  to  design  appropriate  messages.  Team  members 
should  be  advised  on  the  relationship  of  communication  to  relationship  building  within  dynamic,  complex, 
and  stressful  situations.  Indeed,  team  members  need  to  quickly  identify  individual  and  team  information 
needs,  fulfill  the  needs,  and  disseminate,  synthesize,  and  integrate  that  knowledge  into  mission  activities. 
Consequently,  situational  awareness  requirements  can  be  addressed  by  supporting  social  networks  with  access 
to  databases,  human  capital,  and  technology. 
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4.3.6  Future  Challenges 

Teams  play  a  critical  role  in  complex  military  operations  necessitating  multi-operator  environments  [1]. 
Although  one  project  evaluating  teams  in  a  UAV  C2  environment  found  no  performance  differences  between 
co-located  and  distributed  teams,  subtle  differences  in  team  processes  and  individual  task  knowledge  were 
found  suggesting  the  equivalence  in  team  performance  was  achieved  via  modified  team  process  strategies  [2]. 
As  Cooke  et  al.  noted  [2],  “these  results  begin  to  suggest  the  importance  of  considering  long-term  process 
behaviors  that  can  accrue  over  time  ultimately  impacting  team  performance  in  [a]  command  and  control  task”. 
Unfortunately,  the  limited  empirical  evidence  to  date  on  team  performance  in  distributed  environments 
presents  a  significant  challenge  in  itself.  Thus,  the  potential  future  challenges  posed  by  migration  of  operator 
control  in  UMVs  can  best  be  anticipated  by  looking  to  the  current  state  of  knowledge  in  the  fields  of  team 
processes  and  team  communications. 

The  presence  of  interpersonal  and  task  conflict  may  be  perceived  as  negative;  however,  its  relation  to  team 
performance  is  a  positive,  direct  relationship.  For  example,  intragroup  task  conflict  creates  opportunities  for 
team  members  to  better  understand  a  problem  through  team  discussions.  As  a  result,  team  members  are  more 
likely  to  voice  dissenting  or  novel  opinions.  To  balance  and  maximize  the  useful  exchange  of  ideas  in  team 
processes,  it  is  advisable  to  accept  a  moderate  level  of  task  conflict,  but  to  train  members  in  conflict 
management  techniques  or  to  create  a  problem  solving  protocol.  A  moderate  level  of  task  conflict  tends  to  be 
positively  associated  with  team  commitment,  solution  satisfaction,  team  trust,  and  enhanced  performance. 

Interpersonal  conflict  refers  to  value,  goal,  or  need  differences  among  organizational  members.  This  type  of 
conflict  is  frequently  characterized  as  a  personality  incompatibility.  This  negatively  affects  team  trust  and 
performance.  In  this  conflict,  member  focus  is  directed  to  non-productive  behaviors  and  stress  and  emotion 
levels  are  heightened.  Again,  team  training  in  interpersonal  communication  skills  may  be  valuable  for 
avoiding  and  minimizing  this  type  of  conflict. 

Alternatively,  performance  feedback  may  moderate  group  task  conflict,  where  past  issues  involved  in  group 
activities  are  revisited  as  a  result  of  the  feedback.  For  example,  teams  with  positive  relations  tend  to  recover 
and  build  on  negative  performance  feedback.  Teams  with  the  same  performance  feedback  and  a  history  of 
negative  relations,  however,  tend  to  further  degrade  in  quality  of  interactions  and  performance. 

As  part  of  the  restructuring  of  military  hierarchies  to  self-organizing  teams,  it  may  be  advisable  to  move  away 
from  a  process-outcome  view  of  groups.  Peterson  and  Behfar  [3]  suggest  “teams  are  at  particular  risk  of 
experiencing  extremely  high  relationship  conflict  and  poor  future  performance  when  two  conditions  are 
simultaneously  met;  teams  that  do  not  establish  trust  before  they  receive  negative  feedback  are  especially 
vulnerable  to  ongoing  relationship  conflict,  and  likely  to  perform  poorly”. 

Ideally,  team  members  should  manage  the  intensity  level  of  the  conflict.  Balance  the  benefits  of  increased 
critical  reasoning  by  allowing  a  minimal  level  and  avoiding  the  negative  impact  of  destructive  team  interactions 
at  high  intensity  conflict.  De  Dreu  and  Weingart  [4]  “support  the  information  processing  perspective  that 
suggests  that  whereas  a  little  conflict  may  be  beneficial,  such  positive  effects  quickly  break  down  as  conflict 
becomes  more  intense,  cognitive  load  increases,  information  processing  is  impeded,  and  team  performance 
suffers”. 
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Much  of  the  literature  is  based  on  qualitative,  self-report  data.  The  dramatic  consequences  in  the  C2 
environment  require  building  upon  the  findings  of  case  studies  and  building  experimental  evaluation  of  those 
observations  concepts.  Organizational  leaders  need  to  evaluate  the  social  norms,  behavioral  norms,  cultural 
norms  of  team  members,  task  work,  and  team  goals  within  the  context  of  dynamic  environments  to  better 
structure,  support,  and  develop  team  operations.  In  a  virtual  team  environment,  operational  protocols  can  be 
designed  in  accordance  with  the  unique  characteristics  and  requirements  of  the  team  members.  The  behavioral 
tendency  for  a  performer  to  allocate  attention  from  concurrent  tasks  to  the  high  priority  task  to  maintain 
standards  of  performance  [5]  is  valuable  information  for  communication  system  design.  For  example,  verbal 
interactions  among  remote  and  co-located  participants  were  found  to  be  significantly  retarded  when 
participants  were  engaged  in  challenging  tasks  [6].  There  were  no  significant  differences  in  task  SA  and 
performance  for  the  remote  and  in-person  communicators;  however,  verbal  communications  degraded  SA  for 
both  conditions. 

Some  general  recommendations  to  consider  for  future  work  and  research  include: 

•  Conceptualize  work  process  and  structure  as  sequences  of  communicative  actions  which  coordinate 
the  activities  of  members  [7,8]. 

•  Develop  trust  and  cohesion  by  incorporating  face-to-face  meetings  early  in  the  team  creation  process. 

•  Organizational  roles  should  be  clearly  defined  across  team  members/agent  and  task  responsibilities  at 
the  individual  level  (human  and  technology). 

•  Evaluate  which  tasks  are  appropriate  for  virtual  teams. 

•  Tasks  should  be  clearly  communicated  and  unambiguous.  For  complex  tasks,  provide  access  and 
availability  for  increased  communication  and  collaboration. 

•  Create  ways  for  social  ties  to  be  nurtured  in  virtual  teams. 

•  Provide  training  for  team  members  in  communication  skills: 

•  Interpersonal, 

•  Organizational, 

•  Intercultural, 

•  Uncertainty  reduction, 

•  Communication  protocol  development/use,  and 

•  HMI  communication. 

•  Develop  communication  protocols  which  reflect  environmental  contexts,  media  constraints, 
information  complexity,  task  and  socially-related  interactions,  and  technological  capabilities. 

•  Develop  systematic,  empirical  research  programs  to  test  the  behavior  of  human  and  human-machine 
virtual  teams  in  C2  environments.  This  data  is  important  for  determining  appropriate  cultural  and 
structural  changes  to  allow  co-located  and  virtual  teams  to  succeed. 

Additionally,  McCarley  and  Wickens  [9]  reviewed  the  literature  and  summarized  current  UAV  specific 
human  factors  research  shortfalls,  several  of  which  are  directly  applicable  to  the  topic  of  migration  of  operator 
control  in  UMVs: 

•  Develop  and  test  formal  procedures  for  the  transfer  of  control  between  teams  of  operators. 

•  Develop  and  test  HMIs,  automation,  and  procedures  to  ensure  operators  have  adequate  system 
awareness  when  assuming  vehicle  control. 
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•  Delineate  circumstances  under  which  responsibilities  for  vehicle  and  payload  control  may  be  safely 
performed  by  a  single  operator  versus  circumstances  where  responsibility  should  be  distributed  over 
two  or  more  operators. 

•  Delineate  circumstances  where  a  single  operator  can  safely  simultaneously  control  multiple  UMVs. 

The  potential  benefits  and  promise  offered  by  UMVs  in  a  multitude  of  applications  have  captured  the  attention 
of  both  the  military  and  commercial  sectors.  When  technology  changes  rapidly  or  new  and  radical  designs  are 
introduced,  previous  human  factors  data  may  no  longer  be  valid.  It  is  therefore  imperative  to  address  the 
human  factors  and  teaming  issues  arising  from  the  advent  of  migration  of  operator  control  so  the  full  potential 
of  UMVs  is  realized.  It  should  be  obvious  that  rather  than  eliminating  human  factors  concerns,  UMVs  have 
instead  opened  a  new  and  critical  chapter  in  human  factors  research. 
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4.4  MANPOWER  AND  SKILLS 

4.4.1  Selecting  UMV  Crews 

4.4.1. 1  Introduction  and  an  Alternative  Crew  Selection  Method 

As  part  of  System  of  Systems  and  the  Human  Factors  of  Command  and  Control,  personnel  selection  must  be 
considered.  This  assumes  that  humans  will  always  be  part  in  the  command  and  control  of  UMVs  at  some  level 
of  abstraction. 

UMVs  are  new  technologies  for  most  militaries  around  the  world,  and  potentially  require  new  jobs,  positions, 
occupations,  and  units  to  command  and  control  these  assets.  On  the  other  hand,  militaries  have  similar 
manned  vehicles  with  similar  payloads.  The  personnel  that  operate  these  vehicles  are  highly  skilled  and 
knowledgeable,  and  these  skills  and  knowledge  are  potentially  transferable  to  operating  UMVs.  Moreover, 
if  UMVs  were  highly  “intelligent”  or  “autonomous”  then  perhaps  only  general  skill  and  knowledge  levels 
would  be  required  to  operate  the  vehicles  and  their  payload.  The  transfer  of  skills  and  knowledge,  and  the 
requirement  for  general  skill  and  knowledge  levels  will  contribute  to  Force  Multiplication  by  drawing  from  an 
existing,  broader  pool  of  people  that  can  operate  UMVs. 

For  example,  a  pilot,  a  tank  driver,  and  an  Officer  Of  the  Watch  might  be  well  suited  to  fly,  drive,  and  sail 
UAVs,  UGVs,  and  USVs,  respectively.  These  are  positions  with  the  expected  high  level  of  expertise. 
Moreover,  a  navigator,  a  gunner,  and  an  Operations  Room  Officer  may  have  enough  general  skill  and 
knowledge  to  be  able  to  quickly  become  expert  UMV  operators. 

The  Canadian  Forces  Experimentation  Centre  (CFEC)  is  exploring  a  new  method  for  selecting  crews  that 
involve  matching  tasks  and  knowledge  statements  within  a  Canadian  Forces  -  wide  task  and  knowledge 
database  to  predicted  tasks  and  knowledge  statements  derived  from  a  composite  scenario  that  involves  the 
new  technology  -  in  this  case,  multiple  UAVs.  As  more  matches  that  are  made  for  each  position  in  the 
database,  the  more  likely  that  that  position  or  occupation  would  perform  well  as  part  of  the  crew  for  that 
UMV.  This  method  represents  a  substantial  departure  from  common  crew  selection  methods  that  are  often 
based  on  career  progression,  legal  considerations,  availability,  and  ownership  of  the  new  asset.  The  new 
method  has  four  steps  as  follows: 

1)  Decompose  a  composite  scenario  involving  new  technologies  into  a  hierarchy  of  goals. 

There  are  many  techniques  to  perform  mission  analysis  and  function  decomposition  (e.g.,  MIL 
HDBK  46855).  The  technique  used  in  this  case  is  based  on  Perceptual  Control  Theory  applied  to 
function  and  task  decomposition  [1].  The  result  is  a  Hierarchical  Goal  Analysis  (HGA)  where  the 
scenario  is  described  in  terms  of  desired  system  goals.  At  some  level  of  the  hierarchy,  the  goals  are 
assigned  (allocated)  to  humans  and  machines.  Further  research  is  required  to  systematically  develop 
individual  jobs  based  on  predicted  workload  and/or  grouping  tasks  into  roles  and  responsibilities. 

2)  Propose  and  link  the  new  job  elements  (task  and  knowledge  statements)  to  the  goals. 

Job  elements  can  be  associated  with  the  completion  of  those  goals  assigned  to  humans.  The  job 
elements  are  not  limited  to  task  and  knowledge  statements,  but  may  include  the  other  elements 
(such  as  intelligence,  aptitude,  personality,  health,  gender,  etc.)  depending  on  the  level  of  fidelity 
required.  However,  there  are  some  indications  that  current  tasks  may  be  a  first  order  predictor  of  new 
job  performance. 
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3)  Compare  the  new  job  elements  to  the  CF  job  element  inventory. 

This  methodology  for  selecting  crews  depends  on  having  a  task  and  knowledge  statement  inventory 
so  that  the  new  task  and  knowledge  statements  can  be  compared  to  those  in  the  inventory. 
The  mathematical  equation  for  the  comparison  has  been  referred  to  as  a  Job  Similarity  Index  (JSI). 
The  simplest  matching  algorithm,  or  JSI,  is  as  follows: 

jg i  _  w  number  inventory  tasks  that  match  new  tasks  +  w  number  of  inventory  knowledge  that  match  new  knowledge 
2  total  number  of  new  tasks  2  total  number  of  the  new  knowledge 

As  JSI  approaches  one,  then  the  current  job  would  match  the  new  job.  The  hypothesis  is  that  the  job 
performance  is  directly  related  to  the  JSI  value.  That  is,  those  personnel  who  have  a  high  JSI  will 
perform  well  in  the  new  job  and  will  require  minimal  training,  since  they  perform  most  of  the  tasks  in 
their  current  job  and  have  most  of  the  knowledge  from  their  current  job  that  are  required  for  the  new 
job. 

4)  Select  positions  based  on  the  best  match. 


4.4.1.2  Experimental  Design 

CFEC  conducted  an  experiment  to  evaluate  this  crew  selection  method  and  to  determine  whether  JSI  predict 
job  performance.  Military  participants  were  asked  to  participate  in  a  6-hour  experiment  that  involved  the 
command  and  control  of  five  UAVs  in  support  of  an  intelligence,  surveillance,  and  reconnaissance  mission  as 
the  UAV  crew  searched  for  a  terrorist  boat  amongst  14  other  fishing  boats  of  a  similar  classification. 

Each  participant  provided  demographic  information  as  well  as  wrote  the  Wonderlic  General  Cognitive  Ability 
(GCA)  test.  The  participant  performed  a  one-hour  UAV  mission  in  DRDC  -  Ottawa’s  synthetic  UAV 
Research  Test  Bed:  20  minutes  in  each  position  as  the  Vehicle  Operator  (VO),  Payload  Operator  (PO), 
and  Mission  Commander  (MC).  Two  other  experimental  staff  members  played  the  alternate  positions,  thus  a 
crew  of  three  was  formed.  Following  the  first  run,  training  was  conducted  in  the  form  of  a  question  and 
answer  period.  A  proficiency  test  was  administered,  and  then  the  participant  repeated  the  hour  scenario  again. 
The  measured  variables  included  demographic  information,  GCA,  task  completion,  assessed  performance, 
Situational  Awareness,  and  proficiency  test  results,  while  JSI  was  calculated  using  the  above  formula  for  each 
participant. 

The  primary  performance  measures  were  subjective  observations  about  a  number  of  tasks.  Two  observers 
rated  performance  on  the  task  from  one  (low)  to  five  (high).  A  zero  meant  that  the  task  was  not  performed. 
Another  measure  of  performance  was  a  situational  awareness  (SA)  map  tasks  where  the  subject  was  asked  to 
recall  the  friendly,  neutral,  unknowns,  and  unfriendly  radar  returns  and  draw  them  on  a  map.  Their  answers 
were  compared  to  ground  truth,  and  high  scores  were  given  when  the  two  maps  were  similar. 

The  data  analysis  was  designed  to  show  whether  performance  and  JSI  are  related.  Figure  4-5  is  a  fictitious 
example  showing  the  scatter  of  data.  Excel™  was  used  to  calculate  trendlines  as  well  as  the  normalized  mean 
distance  (m.d.)  between  the  data  points  and  the  trendline.  If  the  trendline  has  a  positive  slope, 
then  performance  and  JSI  are  related.  If  the  slope  is  zero,  then  there  is  no  relationship.  If  the  slope  is  negative, 
then  the  two  variables  are  inversely  related.  As  the  mean  distance  approached  zero,  then  the  points  would  fall 
onto  the  trendline  and  there  would  be  high  confidence  that  the  data  and  the  trendline  were  correlated.  As  the 
mean  distance  approached  100%  of  the  maximum  possible  mean  distance  then  there  would  be  low  confidence 
that  the  data  and  trendline  were  correlated.  Note  that  this  is  a  preliminary  analysis  in  order  to  obtain  generic 
impressions  of  the  variables’  relationships. 
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Performance 


Figure  4-5:  Anticipated  Results  Showing  that  Performance  is  Directly  Related  to  JSI. 


4.4.1.3  Results  and  Discussion 

Figure  4-6  shows  the  preliminary  results  in  graphical  form.  The  first  graph  indicates  that  task  completion  does 
not  depend  on  JSI.  That  is,  all  participants  were  able  to  complete  97.7%  of  the  UAV  tasks  regardless  of  the 
percentage  of  tasks  and  knowledge  they  had  in  their  current  occupations.  The  mean  distance  between  the 
trendline  and  the  data  was  1.3%,  which  gives  us  a  very  high  degree  of  confidence  that  the  data  are  correlated 
with  the  trendline. 
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Figure  4-6:  JSI  versus  Performance  Results. 

The  second  graph  indicates  a  slight  positive  relationship  between  JSI  and  the  assessed  performance  results 
(i.e.,  slope  =  +0.13).  However,  our  confidence  that  the  data  and  trendline  are  correlated  is  lower  because  the 
mean  distance  increases  to  10%.  Note  that  the  intercept  value  is  59.9%.  That  is,  even  participants  who  do  not 
perform  any  of  the  UAV  tasks  or  have  any  of  the  required  UAV  knowledge,  still  have  a  general  level  of  tasks 
and  knowledge  that  they  can  achieve  60%  performance.  Anecdotally,  only  after  several  minutes  of  training  on 
the  system,  the  performance  rises  (training  proficiency  test  average  mark  =  74.2%).  Also,  the  other  two  crew 
members  (a  private  and  a  corporal)  became  experts  at  all  the  positions  well  within  3  hours.  Clearly, 
this  simple  composite  scenario  required  a  general  level  of  tasks  and  knowledge  in  order  to  achieve  adequate 
performance.  (N.B.  the  scenario  was  validated  by  experienced  CF  Aurora  crew  members  whose  job  is 
maritime  surveillance  and  reconnaissance). 
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Table  4-1  summarizes  the  relationships  between  JSI  and  other  variables  measured  in  this  experiment,  and 
Table  4-2  summarizes  the  relationships  between  the  variables  and  assessed  performance.  A  detailed  analysis 
would  require  a  multiple  regression  as  well  as  a  test  for  significance.  Also,  one  must  determine  the  minimum 
slope  value  by  which  it  can  be  said  that  JSI  is  related  to  performance,  as  well  as  the  maximum  mean  distance 
that  would  still  indicate  a  high  confidence  level.  Never-the-less,  there  are  indications  that  JSI  predicts 
performance  to  some  degree.  However,  this  scenario  was  simple  enough  that  a  broader  pool  of  participants 
with  general  tasks  and  knowledge  could  complete  97.7%  of  the  tasks  at  a  performance  level  of  least  60%  with 
minimal  training  and  a  fairly  robust  human-computer  interface. 

Table  4-1:  JSI  versus  Measured  Variables 
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Table  4-2:  Measured  Variables  versus  Assessed  Performance 
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4.4.1.4  Concluding  Remarks 

There  are  indications  that  JSI  can  be  used  to  predict  job  performance  to  some  degree.  However,  there  are  other 
factors  that  influence  job  performance  including  the  simplicity  of  the  job  in  combination  with  a  good  human- 
computer  interface.  As  UMVs  become  more  “intelligent”  (i.e.,  operating  at  high  levels  of  automation), 
then  operators  will  only  need  a  general  level  of  job  elements  in  order  to  work  the  systems  at  relatively  high 
performance  levels.  The  results  seemed  to  indicate  that,  depending  on  the  scenario  and  equipment  used, 
one  could  select  from  a  larger  population  of  current  military  personnel. 
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The  alternative  method  for  crew  selection  is  not  tied  to  any  particular  application,  and  is  hoped  to  be  an 
objective  way  of  selecting  crews  particularly  when  no  job  incumbents  exist.  This  would  be  the  case  for  most 
UMV  systems  coming  online.  As  countries  world-wide  begin  to  build  their  UMV  units,  they  will  need  a 
method  for  staffing  the  units  either  with  existing  personnel  or  new  recruits  if  specific  skill  sets  are  not  found 
within  the  organization.  The  method  yields  task  and  knowledge  statements  for  a  given  scenario,  yet  the 
method  itself  can  be  applied  to  any  scenario. 

The  method  requires  an  existing  database  for  the  task  and  knowledge  statement  lexicon  for  step  3.  That  is  a 
common  lexicon  for  tasks  and  knowledge  statements  is  required.  Even  between  environments  (air,  land, 
and  sea)  a  single  task  may  have  several  meanings.  For  example,  secure  a  building  may  mean  surround  the 
building  with  troops  for  army  personnel,  close  all  doors,  windows,  and  hatches  for  navy  personnel,  and  purchase 
or  lease  a  building  for  air  force  personnel.  Initial  discussions  have  started  in  developing  an  Intelligent  Agent 
solution  that  would  help  “translate”  between  environments’  lexicons.  The  next  step  would  be  to  expand  the  tool 
so  that  it  could  “translate”  between  interagency  and  multi-national  task  and  knowledge  lexicons,  if  they  existed. 

This  study  showed  that  the  operator  to  vehicle  ratio  is  moving  in  the  desired  direction  for  Force  multiplication. 
That  is,  the  operator  to  vehicle  ratio  in  this  case  was  3  to  5,  and  the  operators  only  required  a  general  level  of 
task,  skills,  knowledge,  and  training.  In  the  absence  of  incumbents  for  UMVs,  this  crew  selection  method 
promises  to  be  a  viable  alternative  that  would  cross  not  only  environmental  boundaries,  but  also  multi-national 
and  interagency  boundaries.  Thus  the  method  is  compatible  with  system  of  systems  and  interoperability 
thinking. 


4.4.1.5  References 
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4.5  UNINHABITED  MILITARY  VEHICLE  OPERATOR  TRAINING 
4.5.1  Training  of  Decision  Making  Skills  Based  on  Critical  Thinking 

An  important  aspect  of  decision  making  skills  in  real-world  situations  how  to  decide  on  a  course  of  action 
even  though  available  information  may  be  uncertain  or  incomplete,  or  relevant  information  is  simply  missing. 
Since  there  typically  are  no  clear  cut  answers  in  situations  like  this,  any  augmentation  of  the  decision-making 
process  in  the  form  of  training  has  to  focus  on  how  the  decision  makers  interpret  the  information  at  hand. 
For  example,  how  to  consider  the  relevant  factors,  make  plausible  assumptions,  and  identify  conflicts  in  the 
information.  Several  studies  have  shown  that  these  decision-making  processes  can  be  augmented  by 
improving  the  decision  makers’  critical  thinking  for  self-critiquing,  through  training  [1]  and  decision  support 
systems  [2]. 

There  are  several  theories  of  critical  thinking,  but  the  one  of  most  interest  here  is  critical  thinking  as  a 
dialogue  that  has  been  developed  by  Marvin  Cohen  and  associates  in  a  series  of  papers  [3,4,5].  According  to 
their  theory,  critical  thinking  is  a  series  of  questions  and  answers  that  serves  to  investigate  alternative 
positions  of  what  the  available  information  may  indicate.  Three  roles  are  involved: 

1)  The  proponent  who  defend  a  position  by  introducing  more  reasons  that  are  only  consistent  with  the 
current  position; 
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2)  The  opponent  who  asks  for  missing  reasons  or  introduce  rebuttals  that  cancels  reasons  or  their  effect 
on  the  conclusion;  and 

3)  The  facilitator  who  regulates  the  process  in  terms  of  the  relevance  of  reasons  and  whether  the 
dialogue  achieves  the  overall  goals  [4,5]. 

Often,  the  opponent  tries  to  expose  uncertainties  in  the  proponent’s  position  which  may  be  incomplete  when 
some  relevant  reasons  are  missing  [3].  The  reasons  that  the  proponent  introduces  as  a  response  to  the 
incompleteness  may  then  conflict  with  existing  reasons  by  also  supporting  alternative  positions.  One  or 
several  reasons  must  then  be  revoked  or  assessed  in  terms  of  reliability  since  only  one  position  can  be  chosen. 
Thus,  the  critical  dialogue  between  the  proponent  and  opponent  encourages  introduction  of  reasons  that  have 
not  yet  been  considered  by  the  other.  These  reasons  are  then  used  to  assess  positions  in  terms  of  their 
plausibility,  correspondence  between  reasons  and  observations,  coherence  of  reasons,  and  the  uncertainty  of 
the  position  based  on  the  number  of  alternative  possibilities  where  the  position  would  be  incorrect  [5]. 
Consequently,  the  decision  quality  improves  over  time  as  the  partners  learn  more  about  the  positions’ 
strengths  and  weaknesses.  The  critical  dialogue  is  applicable  to  both  team  and  individual  decision-making  by 
switching  between  the  roles  [6]. 

An  important  aspect  of  the  critical  dialogues  is  that  they  can  be  adapted  to  the  available  time  until  a  decision 
has  to  be  made.  There  are  several  dialogue  types,  such  as  persuasion,  inquire,  information  seeking, 
negotiation,  deliberation,  and  eristic  [7],  that  vary  to  which  extent  assumptions  are  questioned.  The  decision 
makers  can  therefore  control  the  amount  of  questioning  and  number  of  possibilities  to  consider  by  choosing 
the  best  dialogue  type  depending  on  the  context.  Since  the  critical  dialogues  uses  the  available  information 
within  the  constraints  of  the  situations,  there  is  no  conflict  with  the  assertiveness  and  rapid  responses  that  is 
often  required  in  military  domains  [4].  Should  the  decision  makers  be  in  severe  time  pressure,  they  may  also 
choose  to  adopt  a  more  recognitional  decision  making  strategy  instead  of  using  critical  thinking  [8]. 
The  ability  to  adapt  the  decision  process  depending  on  the  context  is  often  an  important  aspect  of  expertise. 
For  example,  studies  by  Freeman,  Cohen,  and  Thompson  [9]  and  Cohen,  Adelman,  and  Thompson  [10]  show 
that  experienced  commercial  airline  pilots  adapt  their  decision  process  for  diversion  to  an  alternate  landing 
site  depending  on  the  available  time  and  uncertainty  about  the  need  for  diversion.  Less  experienced  pilots  did 
not  show  the  same  responsiveness. 

Training  of  critical  thinking  attempts  to  encourage  a  dialogue  that  clarifies  disagreements  and  weaknesses  in 
positions.  For  example,  Cohen  et  al.  [5]  describe  a  training  program  that  explains  the  three  roles,  how  to 
perform  a  critical  thinking  dialogue  (adapted  from  von  Eemeren  and  Grootendorst  [11]),  and  general  rules  that 
encourages  a  critical  dialogue.  After  first  developing  their  own  positions,  the  teams  identify  important 
disagreements  and  how  to  resolve  them  by  challenging,  defending,  and  modifying  positions  until  the  parties 
agree  or  there  is  no  more  time  for  discussion.  The  effect  of  the  training  program  was  investigated  using  Army 
officers  that  performed  tactical  decision  making  games.  The  critical  thinking  training  increased  the  generation 
of  new  options  and  the  number  of  reasons  for  positions.  A  similar  type  of  training  program  has  also  been 
evaluated  for  situation  assessment  of  enemy  intent  [1,12],  and  planning  of  a  course  of  action  for  tactical 
decision-making  games  [10].  This  training  is  based  on  the  improving  the  coherency  of  stories  that  explains  the 
available  information.  The  training  consists  of  four  segments  for  story  creation  and  evaluation,  characteristics 
of  task  specific  stories,  how  to  manage  conflicts  and  generate  alternative  interpretations,  and  when  critical 
thinking  is  appropriate.  Mental  strategies,  such  as  the  Devil’s  Advocate  or  the  infallible  crystal  ball,  are  used 
to  encourage  critical  examination  of  positions  and  reasons.  These  critical  examinations  occur  naturally  in  the 
critical  dialogue  due  the  different  roles.  Overall,  the  training  increased  the  usage  of  relevant  information  and 
identification  of  conflicts  and  assumptions  that  required  further  investigation.  These  changes  in  the  decision 
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process  often  improved  the  final  decision.  The  positive  effects  of  critical  thinking  training  are  not  surprising 
since  many  aspects  of  the  training  are  based  on  differences  between  how  experienced  and  less  experienced 
officers  handle  similar  types  of  situations  [12].  Similar  effects  of  experience  have  also  been  found  for  how 
commercial  airline  pilots  evaluate  options  for  diversion  due  to  bad  weather  [9,10].  In  addition  to  adapting  the 
decision  process  depending  on  the  available  time,  experienced  pilots  also  requested  more  information  to 
address  sources  of  uncertainty.  Although  Cohen,  Adelman  and  Thompson  [10]  propose  a  strategy  for  training 
critical  thinking  skills  of  commercial  airline  pilots,  no  training  program  has  been  developed  or  evaluated  for 
option  evaluation,  however.  Finally,  the  training  can  be  administered  using  conventional  presentations  or  self- 
administered  individual  homework  assignments  and  use  advanced  simulation  techniques  for  illustration  of 
scenarios. 

The  preceding  discussion  shows  that  critical  thinking  can  enhance  decision  making  in  many  domains  where 
there  is  only  partial  and  uncertain  information.  It  is  therefore  reasonable  to  expect  that  critical  thinking  will 
also  be  important  for  control  UMVs  when  they  are  employed  tactically  as  a  part  of  or  in  close  cooperation 
with  manned  forces.  While  lower  costs  and  reduced  risks  for  military  personnel  make  UMVs  suitable  for  the 
dull,  dirty,  and  dangerous  missions,  they  also  have  several  characteristics  that  affect  their  operation.  Critical 
thinking  appears  to  be  a  useful  approach  for  assessing  how  these  characteristics  can  affect  mission 
accomplishment  within  the  current  context.  Table  4-3  summarizes  a  brief  review  by  Svenmarck  [13]  of  some 
critical  characteristics  that  may  be  important  for  control  of  UMVs.  One  important  characteristic  of  UMVs  that 
is  often  mentioned  in  the  literature  is  the  lack  of  situation  awareness  from  the  limited  field-of-view  of  sensor 
information  for  control  of  tele-operated  vehicles.  The  result  is  often  spatial  disorientation,  lack  of  awareness 
of  surrounding  obstacles,  and  difficulty  of  interpreting  the  overall  situation.  Further,  tele-operation  for 
navigation  and  control  of  the  sensor  suite  are  in  reality  separate  tasks  that  can  not  be  performed 
simultaneously  [14].  Both  tasks  are  also  hampered  by  the  significant  time  delays  in  communicating  sensor 
information  and  control  commands  [15].  The  combination  of  tracking  difficulty  and  a  preference  for 
endurance  typically  makes  UMVs  slower  than  manned  systems.  Often,  several  operators  are  therefore 
required  to  control  and  manage  the  UMVs  due  to  the  demands  on  attentional  and  cognitive  resources.  Overall, 
however,  the  demands  are  considerably  less  than  when  performing  the  same  missions  manually  without 
UMVs,  which  improves  the  endurance.  The  effect  of  these  characteristics  versus  the  benefits  of  less  risks  and 
efforts  of  military  personnel  can  only  be  assessed  by  the  operator  using  the  available  information  about  the 
situation.  Consequently,  a  reliable  decision  making  process,  such  as  critical  thinking,  is  essential  for 
maintaining  control  of  UMVs. 
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Table  4-3:  Summary  of  Characteristics  that  Affect  how  UMVs  are  Employed  Depending 
on  the  Mission  Context  -  the  characteristics  are  listed  as  a  relative  comparison 
of  performing  missions  with  UMVs  vs.  without  UMVs 


Pros 

Cons 

•  Costs  less  than  military  personnel 

•  Can  perform  more  risky  missions 

•  More  precise  weapons  management 

•  Less  physically  and  mentally  exhausting  for 
operators 

•  Provide  limited  information 

•  Time-delay  in  control  and  sensor  information 

•  Tele-operation  requires  attentional  and  cognitive 
resources 

•  Tele-operation  often  require  more  than  one  operator 

•  Limits  interpretation  of  the  situation 

•  Vulnerable 

•  Have  a  higher  failure  rate 

•  Not  optimised  for  stealth 

•  Can  be  slow  to  deploy 

•  Vehicles  that  are  not  fully  automated  must  remain 
within  radio  coverage  for  control.  Currently,  the  radio 
coverage  is  often  limited,  especially  in  built-up  areas 

•  The  radio  communication  is  vulnerable  to  detection 
and  electronic  warfare 
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4.5.2  Embedded  Training  for  UMV  Crews 
4.5.2.1  Introduction 

Earlier  in  this  SoS  chapter,  it  was  stressed  that  Network  Centric  Capabilities  interweaving  sensors,  humans 
and  decision  aids  will  increase  sources  and  quantities  of  information.  These  advances  will  place  heightened 
cognitive  demands  on  UAV  operators  performing  C2  tasks.  While  the  C2  structure  for  the  future  battle  space 
may  not  be  different  from  current  operations,  the  ability  to  fluidly  push  and  pull  information  over  long 
endurance  missions  is  vastly  improved.  This  poses  new  challenges  for  teamwork  in  UAV  operations, 
particularly  when  it  comes  to  migration  of  operator  control,  team  co-ordination  and  co-operation, 
team  situation  awareness  and  sharing  of  information.  These  challenges  emphasize  the  need  for  novel  training 
concepts  for  UAV  crews. 

One  of  the  concepts  that  emerged  from  social  psychology  is  training  of  team  decision-making  using  the 
‘critical  thinking’  paradigm  [1],  which  was  dealt  with  in  the  previous  paragraph.  In  the  current  paragraph, 
a  human  systems  engineering  approach  to  team  training  is  dealt  with,  proposing  Embedded  Training  (ET)  as  a 
possible  solution.  ET  for  UAV  crews  basically  addresses  the  aforementioned  challenges  encountered  in 
teamwork  by  providing  a  team  training  environment  that  matches  the  conditions  of  the  future  battlefield, 
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using  the  actual  UAV  system  in  combination  with  virtual  features  of  the  environment,  such  as  threats,  targets 
and  other  players. 

During  deployment  of  a  single  tactical  or  strategical  UAV  system  (which  may  contain  more  than  one  air 
vehicle)  for  ISTAR  type  missions  typically  a  team  of  50  personnel  (in  the  order  of  magnitude)  is  involved  in 
the  sustained  operation.  These  personnel  consist  of  operations  personnel  (mission  commanders,  air  vehicle 
operators,  payload  operators,  data  analysts),  maintenance  personnel  and  command  personnel,  who  work  in 
shifts  to  maintain  a  continuous  operation.  For  the  purpose  of  ET  we  will  mainly  concentrate  on  what  we  call 
the  core  team.  The  core  team  consists  of  those  operators  that  are  responsible  for  mission  C2  tasks  and 
planning,  air  vehicle  control,  payload  operation  and  immediate  data  analysis.  More  loosely  defined  we  can  say 
that  the  core  team  is  active  during  the  mission  and  present  in  a  Ground  Control  Station  (GCS)  or  in  a  GCS 
collocated  with  a  data  analysis  cell.  The  core  team’s  primary  purpose  is  guaranteeing  mission  effectiveness 
and  safety. 

Each  of  the  core  team  members  has  a  specific  role  and  a  dedicated  working  position  in  the  GCS  or  data 
analysis  cell.  For  operational  training  of  the  core  team  members  it  may  seem  attractive  at  first  sight  to  be 
attainable  to  train  the  skills,  knowledge  and  attitudes  (SKAs)  for  the  separate  roles,  using  specialized  training 
for  each  role.  However,  the  effectiveness  of  the  core  team  as  a  whole  depends  to  a  large  extent  on  the 
teamwork  of  the  members,  more  than  on  individual  SKAs.  This  is  not  hard  to  understand,  since  the  dynamics 
of  ISTAR  type  missions  require  a  precise  co-ordination  between  mission  planning,  air  vehicle  control, 
payload  operation  and  image  analysis/interpretation.  Therefore  emphasis  should  be  placed  on  training  the 
SKAs  that  are  associated  with  teamwork. 

In  a  typical  scenario,  the  mission  commander  indicates  the  points  of  interest  and  their  priorities,  and  takes  into 
account  the  tactical  situation  and  safety  constraints.  Eventually  this  results  in  a  mission  plan,  consisting  of  a 
flight  plan  and  sensor  plan.  The  air  vehicle  operator  controls  the  air  vehicle  according  to  the  flight  plan 
and  the  payload  operator  controls  the  on-board  sensors  according  to  the  sensor  plan.  Data-analysts 
(photo  interpreters,  SAR  analysts  or  ELINT  analysts)  analyze,  interpret  and  report  the  data  to  further  echelons. 
Since  each  on-board  sensor  has  an  optimum  flight  profile  and  altitude,  the  use  of  a  specific  sensor  has 
consequences  for  the  flight  path  of  the  air  vehicle  and  vice  versa.  During  the  flight  the  operators  have  to  deal 
with  unplanned  situations  such  as  threats  to  the  air  vehicle,  changing  weather  conditions,  military  and  civil  air 
traffic,  targets  of  opportunity,  changes  in  the  target  list  and  ‘sensor-to-shooter’  co-ordination,  including 
communications  with  other  entities  in  the  network.  Consequently,  the  mission  plan  must  be  frequently  adapted 
while  the  air  vehicle  is  in-flight,  again  requiring  co-ordination  between  the  operators. 

Teamwork  is  particularly  important  under  high  task  load  or  in  situations  with  a  large  uncertainty.  Inefficient 
and  ineffective  team  co-ordination  under  those  circumstances  can  severely  impair  mission  success  and  safety. 
It  can  be  expected  that  the  frequency  of  task  load  peaks  will  increase  in  the  future,  since  there  is  clear  pressure 
on  lowering  the  operator-to-vehicle  ratio.  Also,  when  the  UAV  is  part  of  a  time-sensitive  targeting  operation, 
uncertainty  and  high  task  load  are  the  rule,  not  the  exception.  When  the  loop  between  sensor  and  shooter  is 
short,  the  pressure  on  the  crew  in  GCS  can  be  extremely  high,  giving  rise  to  psychological  phenomena  that  are 
unique  to  UAV  operations.  Current-day  warfare,  with  highly  unpredictable  targets,  is  clearly  moving  towards 
shorter  sensor-to-shooter  loops. 

In  the  remainder  of  this  section  we  will  further  discuss  team  training,  specifically  for  UAV  operators.  We  will 
also  present  the  concept  of  embedded  training,  and  provide  a  sketch  of  how  such  a  system  could  be  beneficial 
for  operator  team  training. 
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4.5.2.2  What  is  Embedded  Training? 

For  the  current  purposes  we  define  Embedded  Training  (ET)  as  a  form  of  training  in  which  simulated 
‘entities’  such  as  threats  and  targets  are  fed  into  the  various  avionics  systems  of  an  actual  working  UAV- 
system.  This  allows  training  against  virtual  threats  and  with  virtual  targets.  ET  enables  UAV  crew  in  the 
Ground  Control  Station  (GCS)  to  use  the  system  in  a  situation  where  it  was  designed  for,  while  this  situation 
is  not  available  in  every  day  life.  Thus  providing  capabilities  to  train  UAV  crew  more  effectively  using  the 
real  equipment  in  combination  with  a  partly  simulated  environment. 

An  ET  mission  potentially  provides  the  context  of  a  real  UAV  mission,  but  without  the  need  for  the  actual 
presence  of  mission-critical  entities,  such  as  targets  to  be  observed,  ground  threats,  airborne  threats  or  friendly 
forces.  One  proposed  ET  architecture  presupposes  an  airborne  UAV,  capable  of  ET,  which  maneuvers  through 
a  reserved  airspace  sector.  The  interaction  with  aforementioned  entities  is  simulated  with  on-board  equipment. 
Another  proposed  ET  architecture  does  not  require  an  airborne  UAV,  but  only  a  functional  GCS,  that  is 
capable  of  ET.  Multiple,  fundamentally  different  ET  architectures  are  possible,  and  the  aforementioned 
architectures  are  two  examples.  Although  the  circumstances  and  tensions  of  a  real  mission  will  probably  never 
be  accurately  simulated,  ET  can  come  closer  than  traditional  forms  of  training  by  representing  assets 
in  the  real  environment.  A  UAV  embedded  training  is  however  clearly  different  from  a  manned  fighter  ET: 
UAV  operators  are  physically  located  in  a  GCS.  While  the  air  vehicle  performs  its  ‘dull,  dirty  and  dangerous’ 
tasks  in  the  war  zone  the  operators  are  physically  safe.  Why  ET  in  UAV  then?  A  number  of  arguments  can  be 
given,  including: 

•  ET  provides  increased  training  effectiveness  through  added  immersion,  highly  effective  training 
scenarios  and  team  involvement. 

•  ET  potentially  enables  ‘complete’  team  training,  including  training  of  personnel  involved  in  launch 
and  recovery,  CAOC  personnel,  operators  of  manned  platforms,  logistics,  maintenance,  and,  last  but 
not  least,  data  analysts. 

•  ET  would  also  lead  to  efficient  use  of  costly  flight  time.  Independent  of  the  availability  of  other 
players,  operators  can  go  through  complex  scenarios  with  simulated  entities  in  each  flight.  In  another 
application,  mission  rehearsal  could  take  place  while  loitering  or  en-route  to  operational  area. 

Previous  research  with  fighter  aircraft  has  demonstrated  that  ET  is  a  technologically  and  operationally  viable 
concept  [2,3,4].  Reus  and  Stokkel  [5]  and  Roessingh  et  al.  [6]  investigated  specific  PVI-issues  associated  with 
ET.  Now  R&D  needs  to  focus  on  ET  for  UAV  crews.  If  the  operational  use  and  benefits  for  UAV  operations 
can  be  demonstrated,  it  is  worthwhile  to  require  these  capabilities  for  the  next  generation  of  UAVs.  In  the 
following  sections  the  reader  is  given  a  brief  account  of  some  of  the  principles  underlying  ET. 


4.5.2.3  Why  Combine  Real  and  Simulated  Systems? 

The  ET  system  feeds  additional  entities,  such  as  targets  and  threats,  into  the  mission  system  of  the  UAV 
system.  These  data  are  treated  in  the  GCS  as  if  being  real  data  and  will  be  displayed  as  such.  The  crew  in  GCS 
(supposedly  consisting  of  UAV  operator,  payload  operator,  and  a  mission  commander)  needs  to  interact  with 
these  entities  based  on  their  predefined  roles.  In  the  preceding  section  we  argued  that  ET  is  a  form  of  high 
fidelity  mission  training.  This  is  a  strong  argument  for  serious  consideration  of  implementation.  However, 
some  more  direct  and  quantifiable  advantages  can  be  argued  for  as  well. 

•  In  live  training  without  ET,  e.g.,  on  a  training  range,  physical  entities  are  needed  that  act  as  targets, 
threats  or  friendly  forces.  These  physical  entities  are  costly  to  implement  and  obviously  only  have  a 
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restricted  training  value  for  the  personnel  involved  operating  these  entities.  It  is  clear  that  with 
budgetary  constraints  on  live  training  hours,  ET  will  be  very  cost  effective. 

•  Depending  on  the  chosen  architecture  ET  does  not  need  airspace  or  only  relatively  small  volume  of 
airspace.  In  many  of  today’s  and  future  missions,  UAVs  are  equipped  with  long-range  sensors,  which 
means  that  detection  of  targets,  identification,  electronic  warfare,  and  possibly  weapon  delivery, 
may  all  take  place  at  relatively  long  distances  between  the  players.  Airspace  where  such  skills  as 
electronic  warfare  can  be  trained  in  live  training  is  very  limited,  particularly  in  Europe.  Since  ET 
allows  any  geometry  of  threats,  targets  and  other  players  to  be  simulated,  only  a  very  restricted 
volume  of  airspace  is  needed.  ET  allows  virtual  entities  to  fly  outside  the  designated  area  as  long  as 
the  UAV  remains  within  the  designated  airspace. 

•  Because  the  actual  system  dynamics  are  present  in  ET,  this  form  of  training  is  to  be  preferred  over 
legacy  simulation,  particularly  when  the  crew  has  to  combine  various  skills  and  knowledge  in  an 
integrated  manner,  such  as  co-ordination  and  co-operation,  tactics,  perceptual  skills,  control  skills  and 
attention  management. 

•  UAVs  may  operate  under  control  of  other  platforms,  such  as  Airborne  Early  Warning  platform 
(AWACS).  During  training  however  these  systems  are  not  always  available.  ET  is  able  to  provide  the 
AWACS  control  capabilities  in  case  these  systems  are  not  available.  Thereby  ET  trains  the  crew  the 
necessary  skills  on  a  day  to  day  basis. 

•  Certain  capabilities  of  UAVs  will  not  be  used  during  live  training  due  to  security  or  safety  reasons. 
Specifically  Low  Observerability  (LO)  characteristics  are  expected  to  not  being  used  to  the  full  extent 
during  training  because  of  security  aspects.  ET  allows  training  with  the  LO  characteristics  with  virtual 
systems.  Another  aspect  is  training  with  lasers,  e.g.,  for  target  designation.  Lasers  can  only  be  used  in 
a  specific  environment  due  to  safety  reasons.  ET  allows  the  deployment  of  virtual  lasers. 
This  increases  the  crew’s  readiness  for  real  missions. 


4.5.2  A  The  Architectural  Concept  of  Embedded  Training 

Basic  Architecture 

An  ET  system,  implemented  in  a  UAV  system,  consists  of  three  main  simulation  modules  (Figure  4-7). 

•  First,  the  simulation  management  module  performs  many  functions,  such  as  starting  and  stopping 
training  exercises  and  taking  care  that  all  players  participating  in  the  exercise  have  synchronised 
information. 

•  Second,  the  UAV  simulation  module  stimulates  the  on-board  sensors  and  simulates  the  own  weapons 
and  electronic  warfare  systems. 

•  Third,  the  virtual  world  simulation  simulates  the  virtual  entities  in  the  exercise.  The  virtual  world 
simulation  also  comprises  models  for  the  terrain  over  which  the  exercise  takes  place  and  atmospheric 
conditions. 
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Figure  4-7 


Basic  Architecture  of  an  Embedded  Training  System. 


To  make  the  crew  believe  that  they  are  in  a  real  mission,  the  three  simulation  modules  maintain  an  intensive 
two-way  communication  with  the  UAV’s  standard  mission  system.  The  mission  system  consists  of  modules 
that  handle  sensor  data,  weapon  data,  and  electronic  warfare  data.  Every  time  the  crew  gives  an  input  to  one  of 
the  systems  in  the  GCS,  the  simulation  needs  to  be  updated  and,  as  a  result,  new  simulation  data  need  to  be 
fed  into  the  mission  system.  To  ensure  a  safe  exercise  a  specific  safety  layer  that  safeguards  the  UAV  mission 
system  must  be  developed. 

The  above  described  basic  architecture  is  sufficient  for  the  training  of  engagements  in  which  one  ET  equipped 
UAV  system  is  involved  and  all  other  entities  are  virtual.  However,  in  more  complex  and  realistic  exercises, 
that  is,  beyond  the  single  UAV  operations,  an  airborne  datalink  between  ET  carrying  UAVs  and  an  air-to- 
ground  datalink  between  UAVs  and  ground-based  entities  are  needed  to  ensure  that  all  players  have  matching 
sensor  information. 

A  scenario  for  the  exercise  needs  to  be  prepared  in  advance  on  a  dedicated  GCS.  After  sufficient  verification 
of  the  scenario,  its  digital  representation  can  be  loaded  in  the  ET  carrying  UAV,  either  by  physically  inserting 
a  credit-card-size  memory  card  into  the  system  or  by  datalink.  The  GCS  can  also  be  used  for  debriefing 
purposes. 


4. 5. 2. 4.1  Simulation  Management 

During  the  exercise,  the  simulation  management  module  must  control  all  simulations  as  well  as  the  overall 
course  of  the  exercise.  In  addition,  this  module  manages  the  recording  of  the  exercise  and  performance 
measurement.  Since  complex  exercises  have  many  performance  aspects,  proper  assessment  of  crew 
performance  is  a  major  challenge  to  UAV  crew  training  in  general  and  specifically  to  ET.  The  performance 
aspects  include  the  use  of  sensors  and  countermeasures,  selection  of  tactics,  following  the  mission  routes  and 
selection/assignment  of  targets.  Intelligent  methods  are  needed  to  keep  track  and  analyse  the  crew’s  activities, 
to  categorise  crew  error  and  to  determine  mission  success.  Transfer-of-training  will  increase  when  such 
performance  measures  can  be  singled  out,  both  during  the  exercise  and  in  debriefing. 
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4. 5. 2. 4. 2  UA  V  System  Simulation 

In  the  UAV  system  simulation  a  complete  dynamic  model  of  the  UAV  system  is  included.  UAV  system 
simulation  includes  models  for  the  electronic  warfare  system  and  for  on-board  weapons  (when  present). 
An  important  part  of  the  UAV  system  simulation  is  a  realistic  assessment  of  mission  effectiveness.  System 
displays  in  the  GCS,  such  as  the  radar,  the  radar  warning  receiver  and  target  identification  means  must  interact 
as  if  real  targets  are  present.  These  provisions  make  it  possible  that  missions  can  be  fully  exercised. 

4. 5. 2. 4. 3  Virtual  World  Simulation 

The  virtual  world  includes  the  virtual  entities,  their  electronic  signatures,  weapons  and  dynamic  behavior, 
involving  strategies,  tactics,  maneuvers  and  counter  measures.  Moreover,  the  behavior  of  the  virtual  entities 
has  to  be  in  exact  accordance  with  their  individual  role  (ground-based  or  airborne,  friend  or  foe,  etc.). 


4.5.2.5  Safeguarding  for  Embedded  Training 

When  an  air  vehicle  is  being  flown  in  UAV  operator  training,  safety  is  an  important  issue.  Safety  issues  are 
related  to  the  air  vehicle  itself,  and  to  the  interaction  between  the  air  vehicle  and  its  environment.  Systems  and 
working  procedures  help  the  operators  to  guarantee  a  sufficient  safety  level.  As  an  example,  presentation  of 
air  traffic  in  relation  to  the  air  vehicle  allows  the  operators  to  avoid  collisions  with  manned  or  uninhabited 
aircraft.  The  situation  in  an  embedded  training  scenario  is  somewhat  more  complex,  since  parts  of  the 
environment  are  real,  while  other  parts  are  simulated.  For  example,  an  operator  may  concurrently  see  real  and 
virtual  entities  on  the  same  tactical  display. 

We  clearly  do  not  want  to  compromise  safety  by  introducing  virtual  entities  in  a  scenario;  unsafe  situations  in 
response  to  virtual  entities  are  simply  unacceptable.  Thus,  measures  have  to  be  taken  to  avoid  damage  to  the 
air  vehicle  itself,  for  example  caused  by  collisions  with  air  traffic  or  terrain.  Also,  risks  to  third  parties,  such  as 
manned  military  aircraft,  other  UAVs,  civil  air  traffic,  and  population  on  the  ground,  should  be  minimised. 
Therefore,  just  like  in  a  fighter  aircraft  ET  system,  safeguarding  is  an  important  issue  in  the  design  of  a  UAV 
embedded  trainer.  Also,  a  similar  approach  to  safeguarding  can  be  foreseen.  The  starting  point  is  that  all 
operators  are  at  all  time  aware  that  a  simulation  is  running,  so  continuous  presentation  of  the  simulation  status 
is  a  firm  requirement.  Further,  the  simulation  should  explicitly  indicate  when  it  starts  and  stops.  An  aural 
annunciation  in  the  GCS  is  a  proper  method,  since  it  is  independent  of  the  individual  point-of-gaze  of  the 
operator  team. 

Since  displays  can  contain  both  real  and  virtual  information  at  the  same  time,  operators  should  always  be 
aware  which  information  is  real  and  which  is  virtual.  This  helps  them  to  make  the  appropriate  trade-offs  and 
decisions  during  the  training  scenario.  It  is  clearly  not  desirable  to  perform  a  potentially  unsafe  maneuvre  or 
action  in  response  to  a  virtual  entity,  while  this  may  be  totally  justified  in  an  operational  situation.  A  potential 
implementation  for  symbols  on  a  display  is  to  give  the  virtual  entities  a  dedicated  supplementary  tag.  In  the 
fighter  embedded  training  system  that  was  developed  at  NLR,  this  is  accomplished  by  attaching  a  small  “v”  to 
each  virtual  symbol  on  all  displays  where  they  can  appear.  Naturally  such  information  should  be  designed 
carefully  in  order  to  guarantee  positive  transfer  of  training.  Another  effective  strategy  in  the  design  of  displays 
is  to  give  symbols  related  to  real  entities  a  higher  priority.  This  way,  symbols  related  to  virtual  entities  do 
never  obscure  those  related  to  real  entities. 

More  advanced  means  are  also  possible.  Automatic  monitoring  of  the  air  vehicle  and  its  interaction  with  the 
real  environment  can  prevent  unsafe  situations.  This  can  be  accomplished  by  the  continuous  evaluation  of  a 
number  of  safety  rules  by  the  simulation  itself.  The  simulation  immediately  stops  when  one  of  the  rules  is 
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violated,  that  is,  when  it  detects  an  unsafe  situation.  Naturally  this  should  be  properly  announced  to  the 
operators.  As  an  example,  the  system  can  monitor  that  the  air  vehicle  remains  in  a  temporary  reserved  airspace 
during  the  training.  It  could  also  automatically  detect  potential  collisions  with  real  entities  or  real  terrain. 


4.5.2.6  Choosing  an  Embedded  Training  Architecture 

Various  architectures  are  possible  when  implementing  a  UAV  embedded  training  system,  but  two  distinct 
groups  can  be  identified.  In  the  first  group  an  air  vehicle  is  not  required,  only  a  GCS  is  needed  for  the 
exercises.  In  this  case  ET  is  built  into  the  GCS  and  directly  communicates  with  the  GCS  systems.  The  second 
group  is  fundamentally  different  and  involves  a  flying  air  vehicle  during  the  exercises.  ET  then  communicates 
with  the  on-board  systems  and  (via  the  datalink)  with  the  GCS.  The  simulation  itself  can  be  in  the  air  vehicle 
or  in  the  GCS,  while  it  is  also  possible  to  implement  a  part  of  the  simulation  on  board  and  a  part  in  the  GCS. 
Naturally  this  is  related  to  the  decision  to  involve  a  flying  air  vehicle  in  the  training. 

This  is  not  the  place  to  promote  a  specific  architecture,  but  some  arguments  will  definitely  play  a  role  in  the 
selection. 

•  Training  effectiveness.  Nowadays  simulation  standards  are  high,  but  the  psychological  factors 
involved  in  operating  a  high  valued  asset  with  all  potential  safety  consequences,  can  never  be 
reproduced  in  a  simulator. 

•  Cost  of  flight  hours.  These  costs  are  not  only  due  to  fuel,  but  also  to  the  number  of  personnel  involved 
in  the  training,  complicated  logistics,  and  need  for  air  vehicle  maintenance.  Nowadays,  optimal  use  of 
the  limited  number  of  flight  hours  is  an  important  concern. 

•  External  safety  issues  are  important,  particularly  during  training  operations,  when  the  air  vehicle  is 
over  populated  areas  and  when  sharing  airspace  with  other  vehicles.  Safety  of  the  air  vehicle  is 
important  too.  The  costs  associated  with  the  loss  of  an  air  vehicle  and  collateral  damage  on  the  ground 
can  be  enormous,  also  in  the  ‘public  eye’.  Also,  the  consequences  of  a  collision  with  other  vehicles 
can  be  dramatic. 

•  Security.  Flying  an  air  vehicle  implies  using  a  datalink  and  exposing  the  UAV  system  and  tactics  to 
interested  parties. 

4.5.2.7  Team  Training  to  be  Addressed  with  Embedded  Training 

4. 5. 2. 7. 1  Teamwork 

We  define  teamwork  as  the  seamless  integration  of  specific  skills,  knowledge  and  attitudes  (SKAs)  that  allow 
team  members  to  adapt  and  optimize  their  performance.  Team  characteristics  that  distinguish  teams  from 
small  groups  include  the  following  [7,8,9,10]: 

1)  Multiple  sources  of  information. 

2)  Task  interdependencies. 

3)  Coordination  among  members. 

4)  Common  and  valued  goals. 

5)  Specialised  member  roles  and  responsibilities. 

6)  Task-relevant  knowledge. 
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7)  Intensive  communication. 

8)  Adaptive  strategies  to  help  respond  to  change. 

4.5.2. 7.2  Team  Skills 

Some  team  skills  are  frequently  mentioned  in  the  literature.  For  the  purpose  of  UAV  operations  we  have 
selected  four  of  these  skills.  The  first  one  is  Team  monitoring’,  which  means  mutual  performance  monitoring 
by  team  members,  but  also  mutual  workload  monitoring  and  predicting  each  others’  behavior  [11,12]. 
A  second  frequently  mentioned  team  skill  is  ‘exhibiting  flexibility’  which  includes  adapting  to  novel  and 
unpredictable  situations  [13].  A  third  team  skills  is  exhibiting  team  leadership  or  followership  (depending  on 
the  situation),  which  means  motivating  team  members,  exhibiting  team  initiative,  exhibiting  assertiveness  and 
providing  supporting  behaviors  [14,15,16,10].  Fourth,  we  mention  Team  coordination’,  which  is  the  skill  of 
giving  suggestions  or  criticisms,  but  also  accepting  suggestions  or  criticism,  including  performing  of  self¬ 
correction  [17,18].  Less  frequently  mentioned  skills  include  response  coordination,  coordination  activities, 
resource  distribution,  timing,  interpersonal  coordination,  team  decision-making,  shared  situation  awareness 
[19,20,14]. 

4.5.2. 7.3  Team  Knowledge 

Knowledge  is  difficult  to  define,  but  generally  the  following  building  blocks  are  recognised: 

1)  Declarative  knowledge  (facts  and  concepts); 

2)  Procedural  knowledge:  procedures  and  strategies;  and 

3)  Conditional  knowledge:  principles  and  conditions. 

Examples  of  teamwork  knowledge  in  these  different  categories  are 

1)  Understanding  one’s  own  function  in  the  team; 

2)  Knowledge  of  communication  strategies  such  as  ways  to  give  and  receive  feedback  and  constructive 
criticism;  and 

3)  The  principles  and  conditions  for  creating  and  retaining  a  good  teamwork  atmosphere. 

Teamwork  knowledge  is  typically  acquired  during  education,  dedicated  Team  Resource  Management  (TRM) 
courses  and  similar  specialised  initiatives.  However,  teamwork  knowledge  also  builds  up  through  operational 
experience,  which  provides  insight  in  the  operations,  procedures  and  processes  and  the  knowledge  to  keep 
track  of  the  situation  and  to  ‘read  the  game’. 

Obviously,  there  is  much  more  to  say  about  the  knowledge  underlying  successful  teamwork,  and  how  this 
knowledge  is  acquired.  Literature  on  the  topic  is  abundant.  However,  there  is  an  overlap  between  the 
knowledge  underlying  team  skills  and  the  skills  of  the  team.  Knowledge  that  has  been  effectively  acquired 
will  enable  the  development  of  skills  through  practice.  For  example,  knowledge  about  strategies  on  how  to 
communicate  effectively  may  enable  the  skills  to  communicate  effectively  and  knowledge  of  the  effect  of 
stress  on  teamwork,  allows  the  team  to  recognise  the  symptoms  and  develop  teamwork  skills  for  coping  with 
stress. 

4.5.2. 7.4  Team  Attitudes 

Team  attitudes  are  defined  as  an  internal  state  that  influences  a  team  member’s  choices  or  decisions  to  act  in  a 
particular  way  [21].  Attitudes  toward  teamwork  can  have  a  significant  effect  on  how  teamwork  scales  are 
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actually  put  into  practice  [22].  Positive  attitudes  toward  teamwork  and  an  attraction  to  being  part  of  a  team 
(‘collective  orientation’)  have  been  found  to  enhance  team  processes  and  team  performance.  Some  important 
attitudes  found  in  the  general  literature  are  team  spirit,  team  morale,  belief  in  the  importance  of  teamwork, 
team  cohesion,  shared  vision,  mutual  trust,  collective  orientation  [19,23,24,25,26,9,16,10]. 

4.5.2. 7.5  Types  of  Team  Training 

Although  no  specialized  investigations  have  been  made  with  respect  to  the  types  of  UAV  team  training  that 
could  be  addressed  with  ET,  knowledge  from  related  domains  suggests  that  ET  could  be  applied  for: 

•  Qualification  training  (including  training  for  the  conversion  to  the  specific  UAV); 

•  Day-to-day  routine  training  at  the  operational  unit; 

•  Team  Resource  Management  training;  and 

•  Emergency  Management  Command  and  Control  (EMC2)  training. 

Use  of  ET  would  promote  unity  in  operational  procedures  and  doctrines,  and  be  of  use  to  train  effective 
communication  techniques,  learn  to  overcome  barriers  to  effective  communication  and  awareness  of  strengths 
and  weaknesses  in  personal  communication  skills.  Training  scenarios  could,  inter  alia,  be  based  on  actual 
battlefield  incidents  involving  factors  related  to  teamwork  (e.g.,  during  migration  of  control).  Such  incidents 
with  UAVs  in  which  teamwork  was  a  (contributory)  factor  are  known  and  should  be  reported  carefully  and  in 
detail. 


4.5.2.8  Conclusions 

The  effectiveness  of  the  core  team  operating  a  UAV  system  depends  to  a  large  extent  on  the  teamwork  of  the 
team-members.  During  the  flight  over  the  battlefield  the  team  has  to  deal  with  unplanned  situations.  In  these 
situations,  inefficient  and  ineffective  team  co-ordination  can  severely  impair  mission  success  and  safety. 
For  training  of  the  core  team  in  or  near  the  GCS  (mission  commander,  air  vehicle  operator,  payload  operator, 
data  analysts)  Embedded  Training  provides  increased  training  effectiveness  when  compared  to  live  training  or 
legacy  simulation  training.  ET  would  also  lead  to  more  efficient  use  of  costly  UAV  flight  time,  when 
compared  to  live  training.  A  number  of  additional  advantages  have  been  sketched,  the  extent  of  which 
depending  on  the  specific  architecture  of  the  ET  system.  The  two  basic  architectures  are  (1)  ET  technology 
concentrated  on  board  of  the  air  vehicle,  the  latter  being  in  flight  during  ET,  and  (2)  ET  technology  only  in  the 
GCS,  with  no  role  for  the  air  vehicle  during  ET.  The  first  architecture  is  the  most  innovative  and  technological 
challenging;  particularly  to  safeguard  the  air  vehicle  against  mishaps  induced  by  the  simulation.  Teamwork 
has  been  analyzed  with  respect  to  teams  skills,  team  knowledge  and  team  attitudes.  The  tentative  conclusion 
of  this  paragraph  is  that  these  skills  can  be  addressed  by  the  embedded  training  concept,  particularly  when 
applied  in  the  qualification  and  conversion  training,  routine  operational  training,  and  focused  team  training 
such  as  team  resource  management  training. 
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Within  the  scope  of  this  report  on  “Uninhabited  Military  Vehicles:  Human  Factors  in  Augmenting  the  Force”, 
the  present  chapter  is  dedicated  to  the  involvement  of  the  human  factor  with  the  specific  aspect  of  the 
integration  of  artificial  cognition  in  the  process  of  vehicle  guidance  and  supervision.  In  particular,  the  idea  of 
co-operative  control,  i.e.,  the  co-operation  between  the  human  operator  and  automation,  will  be  addressed. 
Hence,  human-automation  integration  can  be  viewed  from  two  different  standpoints,  each  of  which  facilitating 
the  other.  On  the  one  hand,  the  human  has  to  be  considered  as  the  user  of  technology,  being  the  operator  in  a 
somehow  automated  work  environment,  responsible  for  the  pursuit  of  the  ongoing  processes  and  provided 
with  more  or  less  authority.  On  the  other  hand,  the  consideration  of  human  performance  in  work  processes 
suggests  unique  approaches  to  automation  and  decision  systems  design  for  the  future.  These  approaches  reveal 
the  potential  of  human-like  behaving  machines  (in  the  sense  of  rational  behaviour)  in  certain  given  task 
domains,  even  being  able  to  co-operate,  as  well  as  the  potential  of  a  human-centred  automation,  promising 
significant  performance  advances,  once  introduced  into  a  work  place. 

The  following  sections  will  provide  a  discussion  of  the  human  involvement  aspects  as  named  above  from  a 
conceptual  point  of  view,  to  begin  with.  Further  down,  application  examples  taken  from  current  research  will 
be  illustrated,  covering  different  application  areas  as  well  as  different  perspectives  in  terms  of  human 
involvement. 

Firstly,  the  scope  of  the  discussion  will  be  delimited.  Bearing  in  mind  that  the  following  considerations  shall 
have  the  potential  to  be  applicable  in  the  air,  land,  see,  space  and  underwater  domain  likewise,  it  is  useful  to 
restrict  oneself  to  some  more  specific  field,  in  particular  in  closely  application-related  research.  So,  the 
aviation  domain,  specifically  flight  guidance  and  mission  management  of  military  aircraft,  conventionally 
manned  and  unmanned  likewise,  will  mark  the  vantage  point  of  the  following  discussion.  The  motivation  of 
this  selection  against  the  background  of  the  consideration  of  human  cognition  and  decision-making  will  be 
explained. 

The  next  section  will  provide  a  statement  on  the  current,  i.e.,  the  solution  of  conventional  automation  being 
strongly  influenced  by  the  paradigm  of  supervisory  control.  A  framework  for  the  modelling  of  the  work 
process  and  related  control  levels  will  be  briefly  discussed.  Domain  specific  technology  approaches  will  be 
roughly  structured,  again  considering  flight  guidance  as  an  example. 

Problems  arising  from  conventional  automation  approaches  will  be  discussed  in  the  third  section.  Perspectives 
of  future  automation  and  required  extensions  will  be  introduced.  At  this  stage  the  notion  of  an  Artificial 
Cognitive  Unit  (ACU)  as  part  of  a  work  system  will  be  introduced.  The  required  capabilities  of  such  a 
machine  being  a  team  mate  will  be  estimated. 

The  outcome  of  the  consideration  of  these  required  advances  in  automation  is  to  concentrate  on  the  treatment 
of  human  and  machine  cognition  as  an  inter-disciplinary  approach  based  upon  cognitive  psychology  and 
artificial  intelligence  as  branch  of  information  technology.  This  will  be  the  objective  of  the  fourth  section. 
As  an  interim  result  the  theory  of  the  Cognitive  Process  will  be  introduced  in  this  section. 
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Section  5.5  briefly  gives  some  information  on  realisation  aspects  of  the  Cognitive  Process,  being  the 
underlying  theory  itself.  Creating  a  cognitive  system  according  to  this  theory  requires  the  implementation  of  a 
systems  engineering  framework. 

Section  5.6  will  broaden  the  view  from  which  the  issue  of  artificial  cognition  and  co-operative  automation  has 
been  looked  at  so  far,  by  opening  up  the  podium  for  different,  but  related  perspectives  covering  the  fields  of 
Artificial  Intelligence  methods  evaluation,  knowledge  engineering  and  an  application  in  the  underwater 
vehicle  guidance  domain. 


5.1  SCOPE 

As  already  mentioned  in  the  introduction  the  rather  broad  scope  of  possible  air,  land,  sea,  space  and 
underwater  applications  needs  to  be  narrowed  somehow,  taking  advantage  of  digging  deeper  into  the  specific 
problems  of  one  particular  domain,  finally  providing  beneficial  insight  ready  to  be  adopted  by  other 
application  areas.  In  anticipation  of  the  main  scope  of  this  chapter  the  airborne  application  is  the  choice.  It  is  a 
fact,  that  conventional  automation,  a  term  which  will  be  defined  further  down,  can  be  regarded  as  very 
advanced  in  this  domain.  Modem  electronic  fly-by-wire  systems  enable  an  almost  fully  automatic 
performance  of  an  entire  mission,  as  daily  demonstrated  in  thousands  of  civil  airliner  flights.  Generating  and 
pursuing  a  four-dimensional  flight  trajectory  is  not  a  real  technological  challenge  any  more,  but  provides  a 
very  sustainable  platform  for  further  considerations  to  be  endeavoured  here.  Especially  the  higher  levels  of 
cognitive  performance  involving  problem-solving,  and  decision-making  are  still  mostly  attributed  to  the 
human  operator  acting  as  supervisor  of  a  technical  process. 

In  contrast  to  this,  the  situation,  e.g.,  in  ground  based  application  is  somehow  inverted.  Autonomous  driving  is 
still  a  complicated  issue  (as  observed  during  the  recent  DARPA  grand  challenge),  i.e.,  the  automation  of  the 
lower  guidance  levels  including  the  recognition  of  the  nearest  environment  and  the  resulting  stabilisation  and 
tracking  tasks  are  not  at  all  fully  available  today.  In  fact,  current  research  is  focused  here.  On  the  other  hand  a 
car  navigation  system  supporting  on  the  supervisory  control  level  is  almost  present  in  every  upper  middle- 
sized  class  car.  Virtually  every  tactical  decision  emerging  in  every  day’s  driving  will  already  be  covered. 
Currently  up-coming  so-called  driver  assistant  systems,  which  in  most  cases  correspond  with  the  functions  of 
conventional  aircraft  automation,  call  for  action  in  terms  of  central  co-ordination  of  their  supervision  for 
efficient  operation. 

Again,  the  scope  of  this  chapter  shall  be  the  automation  of  tasks  on  that  supervisory  level.  As  a  result,  systems 
shall  be  enabled  towards  autonomous  task  accomplishment.  This  issue  of  autonomy  will  be  discussed  in  some 
more  depth.  Another  very  important  issue  will  be  the  consideration  of  human-machine  teaming  and 
co-operation.  The  following  sub-sections  outline  a  typical  aerial  warfare  mission  to  serve  as  a  benchmark, 
providing  a  most  interesting  challenge  for  the  concepts  to  be  presented  here  -  i.e.,  the  relevant  scenario  shall 
be  described  mostly  on  a  symbolic  level,  minimising  the  involvement  of  the  processing  of  signals.  The  focus 
shall  be  more  upon  the  logical  relations  between  the  objects,  rather  than  upon  their  physical  properties. 

5.1.1  Typical  Scenario  from  the  Military  Aviation  Domain 

The  benchmark  mission  shall  be  taken  from  the  aerial  warfare  domain.  Figure  5-1  depicts  an  overview  of  the 
scenario  and  the  relevant  objects  of  a  multi-ship  air-to-ground  attack  mission.  The  own  forces  consist  of  the 
airborne  component  covering  different  rolls  such  as  reconnaissance  (RECCE),  suppression  of  enemy  air 
defence  (SEAD)  and  attack.  Furthermore,  a  command  and  control  (C2)  component  might  be  involved, 
possibly  airborne,  typically  ground-based. 
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Figure  5-1:  Scenario  for  Multi-Ship  Air-to-Ground  Attack  Mission. 


The  hostile  forces  consist  mainly  of  two  components,  i.e.,  a  military  target,  fixed  or  moving  and  a  ground- 
based  air-defence  system  represented  by  surface-to-air  missile  (SAM)  sites,  which  can  be  switched  on  and  off, 
and  which  are  to  some  extent  known  during  the  mission  preparation  phase  both  of  which  are  separated  from 
the  safe  territory  by  the  forward  line  of  own  troops  (FLOT).  The  mission  order  requires  the  attack  component 
to  destroy  the  hostile  target.  To  achieve  this,  SAM-sites  temporarily  have  to  be  suppressed  or  destroyed. 

Although  massively  simplified  with  respect  to  asymmetric  warfare  scenarios  currently  discussed  by  NATO, 
this  scenario  bears  a  great  variety  of  challenges  in  terms  of  integrated  mission  systems  and  automation. 

5.1.2  Forces  Structure 

The  own  airborne  forces  will  be  a  whatsoever  mix  of  manned  and  un-manned  platforms,  to  begin  with. 
The  scenario  envisions  a  set  of  platforms,  which  have  no  static  role  or  task  allocation  (A  static  role  allocation  in 
this  context  could  be  “Reconnaissance  A/C  or  UAV  searching,  combat  A/C  or  UAV  shooting”).  These  platforms 
form  a  heterogeneous  team,  which  means  that  the  entities  may  differ  from  each  other  with  respect  to  resources 
and  capabilities,  such  as  sensors,  actuators,  weapons,  and  information  processing.  This  heterogeneous  team 
structure  does  not  prohibit  homogeneous  sub-structures,  i.e.,  that  some  team  members  have  equal  or  partially 
overlapping  resources  and  capabilities.  The  envisioned  scenario  requires  co-operation  capabilities  of  the 
participating  forces,  because  otherwise  the  mission  cannot  be  accomplished  [1]. 

A  more  generalised  standpoint  is  shown  in  Figure  5-2.  Starting  from  a  classical  situation,  where  single  or 
multiple  manned  vehicles  perform  the  mission.  The  critical  questions  arise  when  un-inhabited  aerial  vehicles 
(UAV)  enter  the  scene.  It  has  to  be  decided  whether  the  UAVs  will  substitute  or  supplement  the  conventional 
manned  platforms  [2].  Obviously  in  some  cases  substitution  will  be  an  isolated  solution,  especially  thinking  of 
the  so-called  DDD-missions  (dull-dirty-dangerous)  -  but  generally,  we  certainly  have  to  face  the  technological 
challenges  of  the  solution  of  supplementation  of  forces,  including  the  issues  of  manned-unmanned  teaming, 
co-operation  and  supervision. 
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Figure  5-2:  Possible  Characteristics  in  Future  UAV  Deployment - 
Substitution  and/or  Supplementation. 


5.1.3  References 
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5.2  THE  WORK  PROCESS  AND  CONVENTIONAL  AUTOMATION’S  SOLUTION 

The  last  section  gave  a  brief  outline  of  the  challenge  for  future  mission  systems.  Needless  to  say,  this  type  of 
mission  can  already  be  performed  today,  in  one  or  the  other  way.  The  scope  of  this  report  of  course  is  the 
augmented  exploitation  of  presently  unrevealed  abilities  in  manned-unmanned  teaming.  To  do  so,  the  first 
step  here  shall  be  characterisation  of  current  automation,  i.e.,  the  solution  of  conventional  automation.  For  the 
later  discrimination  between  automatic  and  autonomous  performance  the  consideration  of  the  work  system 
will  be  helpful. 

5.2.1  The  Work  System 

The  work  system  as  a  general  ergonomics  concept  [1]  has  been  utilised  in  the  application  domain  of  human- 
machine  co-operation  in  aircraft  flight  guidance  by  [2].  Figure  5-3  shows  an  adaptation  of  the  concept 
incorporating  some  application  specific  imagery  for  the  purpose  of  intuitive  understanding. 
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Figure  5-3:  Concept  of  Work  System. 

The  work  system  consists  of  three  major  elements,  i.e.,  the  operator,  the  work  object  and  operation-assisting 
means,  as  characterised  in  some  more  detail  here: 

•  Operator:  In  the  traditional  view  of  a  work  system  the  operator  is  usually  a  human  operator,  being  in 
charge  of  performing  a  certain  given  task,  such  as  accomplishing  a  combat  mission,  as  according  to 
the  chosen  application.  The  human  operator  is  the  high  end  decision  element  of  the  work  system. 
He  determines  and  supervises  within  the  work  system  what  will  happen  with  the  work  object. 
This  can  be  done  by  working  on  any  required  performance  level,  including  manual  control.  In  highly 
automated  work  systems,  as  we  are  talking  of,  the  human  performance  is  usually  focused  on 
supervisory  control,  including  decision-making  and  problem-solving  in  order  to  comply  with  the 
work  task.  As  according  to  the  common  view  of  ergonomics,  the  abilities  of  the  skilled  and  trained 
human  operator  in  terms  of  information  processing  performance  can  be  seen  as  pretty  much  invariant 
in  an  average. 

•  Work  Object:  The  notion  of  the  work  object  is  not  necessarily  restricted  to  the  physical  nature  of 
whatever  machine,  but  also  comprises  dynamical  processes,  i.e.,  the  progression  of  the  situation  over 
time.  In  the  chosen  application  domain,  the  work  object  may  be  the  mission  of  a  combat  aircraft  or 
UAV. 

•  Operation- Assisting  Means:  The  concept  of  the  operation-assisting  means  can  be  seen  as  a  container 
for  whatever  tools  or  automation  of  the  work  place  is  available,  being  computerised  pieces  of 
technology  in  many  cases.  In  our  application  domain  an  auto-flight/autopilot  system  including  the 
human-machine  control  interface  (i.e.,  FCU  -  flight  control  unit),  or  even  the  aircraft  itself  as  a  means 
of  transport  may  serve  as  typical  examples.  Common  to  the  nature  of  various  operation-assisting 
means  is  the  fact  that  they  only  perform  certain  sub-tasks  (e.g.,  pursuing  a  given  flight  trajectory, 
holding  a  defined  heading).  Such  a  sub-task  does  not  form  a  work  system  itself,  obviously  being  only 
a  part  of  another  higher  level  work  task.  In  today’s  common  ergonomic  design,  the  operation-assisting 
means  are  typically  subjected  to  the  endeavours  of  adaptation  and  optimisation  in  order  to  meet 
overall  system  requirements  and  further  improvements. 
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These  elements  will  be  combined  to  the  work  system  set  up  in  order  to  achieve  a  certain  work  result  on  the 
basis  of  a  given  high  level  work  task.  The  accomplishment  of  a  military  flight  mission  may  give  a  good  idea 
of  what  is  meant  here.  Finally,  environmental  conditions  and  external  resources,  such  as  information,  material, 
or  energy  will  affect  the  ongoing  work  process. 

The  concept  of  the  work  system  seems  very  suitable  for  the  consideration  of  problems  to  be  discussed  in  the 
further  pursuit  of  this  elaboration.  The  reason  for  this  is  the  fact  that  the  work  process  is  constituted  by  the 
work  task  and  the  desired  result,  no  matter  its  technical  or  organisational  structure.  However,  exactly  this 
technical  or  organisational  structure  might  as  well  be  easily  modelled  and  analysed  by  the  framework  given  by 
the  work  system.  Yet,  the  notion  of  the  work  system  at  this  stage  gives  no  hints  of  modelling  the  mechanisms 
of  human  performance. 

In  order  to  do  so,  a  very  common  model  of  human  control  performance  shall  be  mentioned  here,  where  a 
distinction  is  drawn  between  manual  and  supervisory  control.  This  issue  has  been  elaborately  investigated  by 
Thomas  B.  Sheridan  at  MIT  (e.g.,  [3])  with  a  more  recent  focus  on  tele-operation  [4],  where  obviously 
supervisory  control  predominates  due  to  the  remoteness  of  the  work  object. 

Figure  5-4,  which  is  adapted  from  [3],  shows  an  automated  human-machine  system  with  the  human  operator 
in  manual  control  mode  on  the  left  hand  side.  In  this  situation  the  human  operator  is  busy  in  feedback  control 
of  the  inner  loops  of  the  underlying  process.  Typical  for  the  manual  control  mode  are  any  kind  of  tracking 
tasks  such  as  lateral  car  steering  or  attitude  control  of  an  aircraft.  Automation  is  mainly  responsible  for  the 
transformation  and  transmission  of  the  required  signals.  On  the  right  hand  side  of  Figure  5-4,  the  automation 
takes  over  the  role  of  automatically  closing  higher  bandwidth  control  loops  as  to  the  dynamic  process.  In  this 
case  the  human  operator’s  role  is  shifted  towards  the  supervisory  control  mode,  where  the  tasks  of  monitoring 
and  setting  demand  values  for  the  automated  control  process  are  relevant. 


manual  control  supervisory  control 

Figure  5-4:  Manual  and  Supervisory  Control. 
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In  order  to  approach  another  definition  of  supervisory  control  [5]  states: 

“When  a  process  is  semi- automated  or  responds  very  slowly,  it  is  not  necessary  for  a  human  to 
devote  full  attention  to  that  process,  [...]  In  situations  where  [...  the  process]  is  automatically 
controlled,  we  can  view  the  human  as  a  supervisor  whose  role  includes  monitoring  the  process 
[...],  adjusting  the  reference  points  [...],  and  intervening  in  the  case  of  failures  and 
emergencies.  ”  [Rouse,  1980] 

Many  real-world  applications  in  fact  will  require  human-machine  interaction  as  a  mixture  of  manual  and 
supervisory  control  as  a  function  of  the  level  of  automation  selected.  The  human  operator  will  permanently 
toggle  between  the  two  control  modes,  allocating  varying  amounts  of  attention  to  one  or  the  other  task. 

In  order  to  prepare  a  common  ground  for  the  further  discussion  of  models  of  human  performance,  this  sub¬ 
section  shall  be  closing  with  the  introduction  of  a  human  model  of  manual  and  supervisory  control  advocated 
by  [5].  The  adapted  model  is  set  in  the  aviation  domain  context  and  depicted  in  Figure  5-5. 


Figure  5-5:  Model  of  Human  Manual  and  Supervisory  Control. 


In  Figure  5-5  the  direct  functional  chain  of  sensing  process  and  environmental  parameters,  filtering  the 
information,  applying  control  laws,  and  finally  acting  on  the  process  represents  all  that  is  involved  in  the 
execution  of  manual  control.  On  a  supervisory  control  level  gathered  and  filtered  information  will  be  fed  into 
a  functional  block  representing  problem-solving,  planning  and  decision-making.  This  block  in  turn  will 
determine  the  demand  values  for  the  controller.  Furthermore  it  allows  the  selection  of  the  control  mode  and 
the  adaptation  of  the  control  laws  according  to  the  current  task.  Finally,  the  decision-maker  will  adjust  the 
filter  in  terms  of  selective  allocation  of  resources  such  as  attention  (e.g.,  [6]). 

5.2.2  The  Hierarchy  of  a  Conventional  Guidance  and  Control  System 

In  the  previous  sub-section  the  focus  was  drawn  to  the  human  operator’s  aspects  of  the  work  system  for  the 
first  time.  This  sub-section  shall  concentrate  more  upon  the  operation-assisting  means,  i.e.,  the  automation. 
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Obviously,  the  characteristic  of  the  operation-assisting  means  is  dependent  on  the  application  domain  to  a 
great  extent.  This  is  the  point  where  we  get  back  to  the  aviation  domain  as  an  example. 

Figure  5-6  shows  the  major  building  blocks  of  a  common  hierarchical  architecture  of  a  state-of-the-art  flight 
guidance  and  control  system  with  several  nested  loops  (adapted  from  [7]).  Besides  the  many  closed  control 
loops  on  the  machine  side,  one  loop  is  closed  involving  the  human  operator,  i.e.,  the  pilot.  Obviously  this 
architecture  puts  the  pilot  into  a  versatile  situation  of  supervisory  control,  using  all  these  fancy  machine 
functions.  Direct  intervention  in  manual  control  style  is  likewise  possible  on  the  lowest  (i.e.,  most  right  in 
Figure  5-6)  interaction  level.  It  should  be  mentioned  that  Sheridan’s  notion  of  manual  control, 
being  unaffected  by  automated  control  loops,  is  to  some  extent  impaired  by  the  current  technology  of  control- 
configured  vehicles  (“fly-by-wire”),  where  the  lowest  available  human  interaction  level  already  implicates 
automatic  control.  Nevertheless,  the  human  interaction  with  such  a  system  might  be  denoted  as  manual  control 
on  that  particular  interaction  level. 


While  the  pilot  is  controlling  and  supervising  his  machine,  he  himself  is  supervised  by  some  external  authority 
such  as  any  imaginable  implementation  of  command  and  control.  In  many  western  leaderships,  the  interface 
between  command  and  control  and  the  local  operators  is  implemented  on  the  basis  of  the  assignment  of  work 
orders,  i.e.,  mission  orders  in  our  domain. 

A  major  performance  feature  of  an  educated,  trained,  and  well  skilled  operator  is  the  capability  of 
transforming  this  work  order  into  a  desired  work  result.  This  structure,  though,  is  tightly  related  to  the 
conception  of  the  work  system  according  to  the  previous  sub-section. 

Figure  5-7  shows  a  situation  which  emerges  when  the  pilot  is  removed  from  the  vehicle  and  placed 
somewhere  else,  e.g.,  in  a  ground  control  station.  Again,  this  remote  operator  will  receive  a  mission  order 
from  any  superior  command  and  control  authority.  Usually,  the  operator  now  will  interact  with  the  UAV  by 
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passing  a  detailed  mission  plan,  which  has  to  be  worked  out  on  the  basis  of  the  mission  order,  to  the  vehicle. 
In  the  case  of  a  fully  automated  system,  this  initial  mission  plan  will  be  pursued  by  the  vehicle  by  use  of  the 
available  on-board  technology.  Usually,  with  conventional  technology,  exceptional  situations  on  the  mission 
level,  such  as  occurring  obstacles,  changes  in  the  tactical  situation,  or  other  constraining  factors  cannot  be 
handled.  As  a  result  of  a  monitoring  function  of  the  ground  operator  adaptations  of  the  mission  plan  or 
reversing  to  outer  loop  guidance  commands  may  occur.  Usually,  there  are  a  couple  of  restraining  factors  for 
the  remote  operation  of  the  vehicle: 

•  Manual  control  of  the  inner  loops  may  not  be  possible  or  desirable  because  of  intolerable  time  delays 
in  the  data  transmission  with  respect  to  the  inner  loop  dynamics  time  constants.  Thus,  the  remote 
operation  heavily  relies  upon  the  availability,  the  performance  and  integrity  of  some  specific  guidance 
functions,  such  as  auto-land,  otherwise  requiring  manual  interactions. 

•  Insufficient  downlink  bandwidth  and/or  incomplete  sensor  coverage,  with  respect  to  the  task, 
can  cause  what  may  be  called  “keyhole  perspective”  [8]  for  the  remote  operator,  potentially  affecting 
the  correctness  or  quality  of  his  or  her  decisions. 

•  The  availability  of  data  link,  i.e.,  the  ability  to  monitor  (via  telemetry)  or  control  (via  telecommand) 
the  vehicle  remotely  may  be  disturbed.  As  a  result,  no  recognition  of  nor  reaction  to  unexpected 
situations  is  possible  any  more  on  the  human  operator’s  side. 


What  just  has  been  elaborated  for  the  flight  guidance  and  navigation  task  holds  true  for  other  concurrent  tasks 
of  the  operator,  such  as  responding  to  a  tactical  environment  or  deploying  mission  related  payload,  as  well. 
Air-to-air  combat  may  serve  as  an  extreme  example,  where  sensory  information  from  radar  and  identification 
equipment  dictate  the  operator’s  actions  with  regard  to  trajectory  determination  as  well  as  weapon  aiming  and 
deployment,  altogether  facilitated  by  complex,  highly  automated  systems  themselves.  Here  again  a 
complicated  mixture  of  manual  and  supervisory  control  tasks  can  be  observed.  Automation  technology  is 
predominantly  available  on  the  manual  control  level,  if  at  all. 
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Figure  5-8  tries  to  summarise  the  just  now  characterised  situation  with  respect  to  conventional  automation. 
In  order  to  achieve  a  desired  work  result,  running  a  machine  or  controlling  a  process,  usually  a  more  or  less 
wide  spectrum  of  tasks  and  related  sub-tasks  has  to  be  worked  on.  This  may  include  sub-tasks  such  as  flying 
an  aircraft,  operating  in  a  tactical  scenario,  managing  avionics  systems,  and  communicating  with  others,  each 
of  which  involving  automation  to  some  specific  extent.  Although  there  may  be  “horizontal”  interaction 
between  automation  involved  in  different  task  domains  to  some  limited  extent  (e.g.,  the  automatic 
performance  of  a  terrain  evasive  manoeuvre,  or  the  automatic  transmission  of  radar  tracks  via  tactical  data 
link),  the  integration  of  information  in  order  to  pursue  the  overall  task  is  performed  by  the  human  operator  on 
a  supervisory  control  level  mostly.  So,  the  interaction  within  the  automation  is  predominantly  vertically 
structured.  A  conventional  flight  guidance  and  control  system  (see  Figure  5-6)  is  certainly  a  very  good 
example,  whereas  the  human  operator  is  supposed  to  toggle  between  the  different  tasks  horizontally  on  a 
supervisory  performance  level. 


Figure  5-8:  Organisational  Structure  of  Conventionally, 
i.e.,  Hierarchically  Automated  Human-Machine  Systems. 

Having  this  rather  simple  organisational  model  of  automation  at  hand  the  following  section  shall  illuminate 
some  technical  pitfalls  associated  with  this  structure  before  some  suggestions  of  improvements  will  be  made. 
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5.3  PROBLEM  DEFINITION 

The  last  section  introduced  one  possible  approach  to  how  automation  in  human-machine  systems  could  be 
looked  at.  Without  being  too  specific  on  particular  mission  systems,  some  peculiarities  of  current, 
i.e.,  conventional  automation  systems  have  been  deduced.  This  section  provides  a  closer  look  upon  problems 
which  may  arise  in  use  of  this  automation  approach.  In  the  further  pursuit  of  this  section  a  possible  perspective 
of  future  automation  technology  will  be  given,  finally  ending  up  with  some  very  particular  requirements  to  be 
implemented  before  such  systems  will  be  put  into  work. 

5.3.1  Shortfalls  with  Conventional  Automation 

It  has  long  since  been  known  that  erroneous  human  action  is  the  predominating  factor  in  aviation  accidents, 
however,  it  is  fair  to  state  that  many  of  these  human  errors  are  caused  by  over-demands  (see  grey  line  in 
Figure  5-9)  on  the  pilot’s  resources  [1],  the  latter  representing  the  natural  limiting  factor  for  performance 
(see  straight  blue  line  in  Figure  5-9).  In  order  to  overcome  this  situation  the  introduction  of  automation  as 
described  above  was  most  beneficial  in  many  situations,  which  otherwise  could  not  be  handled  (see  green  line 
in  Figure  5-9).  On  the  other  hand,  new  types  of  latent  overtaxing-prone  situations  appeared  with  the  increased 
introduction  of  automated  functions  [Onken,  1999]  (see  red  line  in  Figure  5-9). 


Operator 

Workload 


Negative  Effect 
of  Automation 


Demand  on  Operator  Resources 
without  Automation 


Available  Operator  Resources 


Demand  on  Operator  Resources 
with  conventional  Automation 


Time 


Figure  5-9:  Operator  Overload  Caused  by  Conventional  Automation. 
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Charles  E.  Billings  investigated  typical  shortfalls  of  current  aviation  automation  [2],  with  a  particular  view 
upon  the  human  interaction  with  automation.  According  to  Billings,  the  most  critical  design  factors  are 
complexity  (Will  the  extent  of  the  automatic  function  be  fully  understood  by  the  human  operator?),  brittleness 
(Will  the  complex  automation  be  fit  for  any  imaginable  situation  or  purpose?),  opacity  (Will  the  automatic 
execution  provide  sufficient  and  intelligible  feedback  to  the  human  operator?),  and  literalism  (Will  the 
automation  understand  the  human  operator’s  control  actions  as  ‘naturally’  as  they  are  meant?).  Generally 
spoken,  Billings’  answer  to  these  questions  with  respect  to  current  automation  is  “No”,  resulting  in  a  situation 
which  is  usually  referred  to  as  clumsy  automation  [3]. 

Figure  5-10  explains  the  situation  by  use  of  the  organisational  structure  of  conventional  automation 
with  respect  to  task  allocation  between  automation  and  the  human  operator  as  introduced  previously 
(see  Figure  5-8).  Obviously,  the  classical  task  allocation  suffers  from  some  typical  difficulties  [23]. 


Machine  /  Process 


Tasks 


Machine  /  Process 


Figure  5-10:  Shortfalls  with  Conventional  Automation. 


In  particular  under  the  assumption  of  increasing  complexity  of  automation,  the  human  operator  is  almost 
completely  separated  from  the  underlying  process.  The  long  term  problem  of  loss  of  skills,  i.e.,  erosion  of 
competence,  in  supervisory  control  has  been  widely  reported  on,  e.g.,  [4,5].  Within  the  same  class  of  difficulties 
the  human-out-of-the-loop  problem  represents  the  corresponding  short  term  issue,  addressing  situations  where 
operators  almost  fully  rely  upon  the  automation  performance  to  an  extent  that  any  abnormal  situation  will 
inevitably  cause  human  overload  and  erroneous  action.  [6]  states: 

“[...]  by  taking  away  the  easy  parts  of  his  task,  automation  can  make  the  difficult  parts  of  a 

human  operator's  task  more  difficult.  ”  [6] 

Quite  closely  linked  with  Billings’  notion  of  brittleness  is  the  perception  that  conventional  automation  will 
usually  not  be  able  to  recover  from  undesired  situations  induced  by  malfunctions,  faulty  operations  or  just  the 
unexpected.  The  major  reason  for  this  limpness  of  the  system  is  its  lack  of  excellence  in  situation 
understanding  and  goal  driven  performance  on  the  machine  side,  i.e.,  the  missing  capability  of  current 
automation  systems  to  perform  on  a  supervisory  level  in  order  to  pursue  the  overall  goals  of  the  work  system. 
As  a  good  explanation  for  this  circumstance  the  example  of  a  simple  autopilot  function  may  serve  [24]. 
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Once  activated,  an  “altitude  acquire”  function  will  pursue  its  specific  sub-task  of  capturing  a  flight  altitude 
pre-selected  by  the  pilot  in  an  almost  perfect  manner,  no  matter  what  may  be  of  any  relevance  otherwise, 
e.g.,  ground  or  traffic  proximity,  exposure  to  enemy  radar,  or  faulty  demand  setting  or  mode  selection  by  the 
pilot  in  the  sense  of  for  instance  a  misinterpreted  ATC  clearance.  So,  automation  offers  a  dedicated  set  of 
more  or  less  independent  functions,  each  of  which  being  responsible  for  a  particular  sub-task.  The  situation 
can  get  even  more  precarious  when  these  functions  start  getting  linked  horizontally  without  that  being 
transparent  to  the  human  operator  (i.e.,  opacity  due  to  [Billings,  2]).  Modem  flight  management  systems  often 
bear  this  characteristic,  but  still,  conventional  automation  is  not  at  all  capable  of  performing  any  higher 
decision  loop  in  the  sense  of  supervisory  control. 

5.3.2  Perspectives  of  Future  Automation 

As  an  essence  from  the  last  sub-section,  automation  complexity  can  be  seen  as  the  most  critical  issue. 
To  begin  with,  complex  automation  used  to  be  the  key  to  a  major  increase  in  mission  effectiveness  and  flight 
safety  (see  Figure  5-11).  Due  to  limited  resources  and  capabilities  on  the  human  operator’s  side,  a  further 
increase  of  automation  complexity  has  no  longer  been  beneficial  in  terms  of  these  productivity  factors 
(see  Figure  5-11).  Obviously,  automation  became  too  complex  to  be  reliably  handled  by  human  operators. 
The  reason  for  this  seems  to  be  found  in  the  unpredictability  of  the  machine’s  behaviour  due  to  inconsistencies 
between  the  machine  function  and  the  human  operator’s  mental  model  of  it.  Conventional  automation  itself, 
in  the  first  place  meant  to  be  an  operation-assisting  means,  became  a  complex  element  within  the  already 
complex  work  system. 


In  order  to  tackle  this  problem  a  new  approach  to  automation  has  to  be  introduced  into  work  systems. 
Figure  5-11  illustrates  the  vision  of  further  increasing  the  productivity  factors  effectiveness  and  safety  by 
advanced  automation  at  the  cost  of  furthermore  complexity,  but  how  shall  this  “advanced  automation” 
be  shaped? 

Figure  5-12  tries  to  illustrate  some  first  ideas  in  order  to  overcome  the  problems  with  conventional  automation 
described  earlier.  Advanced  automation  shall  not  displace  the  human  operator  in  a  work  system,  but  share  the 
tasks  in  a  close-partner  work  relationship.  Task  allocation  shall  not  be  static,  but  may  be  adapted  to  the  current 
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situation’s  needs.  This  includes  the  facilitation  of  redundancy  in  functions  in  principal  by  at  least  a  partial 
overlap  in  capabilities  with  respect  to  the  task  spectrum.  The  responsibility  of  automation  (not  necessarily 
authority)  shall  be  extended  to  the  supervisory  control  level,  i.e.,  automation  shall  be  enabled  to  perform 
certain  tasks  under  consideration  of  the  overall  work  task  of  the  work  system.  Thereby,  particularly  brittleness 
will  be  tackled.  Coordination  and  communication  with  such  an  automation  system  shall  be  supported  on  all 
performance  levels,  i.e.,  reaching  from  detailed  low  level  information  (reducing  opacity  of  the  machine 
solutions)  up  to  abstract  human-like  information  exchange  on  the  supervisory  level  (tackling  literalism  of  the 
automation).  In  general,  it  may  be  accepted  that  this  approach  to  cognitive  coupling  [7]  can  be  a  contributing 
factor  to  the  mitigation  of  disadvantageous  complexity  effects. 


Figure  5-12:  Co-operative  Structure  of  Human-Machine  Systems  with  Advanced  Automation. 


5.3.2.1  Cognitive  Automation 

An  entity  enabled  to  exhibit  the  aforementioned  behaviour  facets  shall  be  referred  to  as  Artificial  Cognitive 
Unit  (ACU)  [24].  As  indicated  above,  supervision  and  co-operation,  as  accomplishments  of  a  machine  system, 
require  special  capabilities.  These  capabilities  were  combined  within  the  notion  of  such  an  Artificial  Cognitive 
Unit.  Obviously,  the  performance  feature  of  cognition  is  the  core  element  which  has  to  be  dealt  with  in  order 
to  design  such  an  ACU.  From  the  point  of  view  of  the  discipline  of  cognitive  psychology  (e.g.,  [8,9])  human, 
i.e.,  natural  cognition  can  be  described  by  considering: 

•  Perception  and  allocation  of  attention; 

•  Knowledge  representation  and  memory; 

•  Problem  solving,  reasoning  and  decision  making; 

•  Language  comprehension  and  its  generation;  and 

•  Learning  and  the  development  of  expertise. 

The  availability  of  at  least  some  of  these  aspects  of  cognition  are  the  necessary  pre-requisite  to  perform  the 
supervisory  control  task  (compare  Sheridan,  [26])  with  respect  to  the  compliancy  with  the  overall  work  task. 
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Figure  5-13  shows  the  work  system,  according  to  Figure  5-3,  with  the  human  operator  mimicked  by  an  ACU. 
In  this  configuration  the  ACU  represents  all  the  performance  requirements  found  to  be  attributed  to  the  human 
operator  earlier  on,  i.e.,  the  performance  of  decision-making,  problem-solving  and  supervision  of  the  operation- 
assisting  means  and  the  work  object  in  order  to  comply  with  the  overall  work  task.  The  major  difference  is  that 
the  ACU  is  no  longer  invariant  in  terms  of  performance  characteristics  like  its  human  archetype,  but  on  the  other 
hand,  there  has  to  be  found  a  way  how  to  design  it  according  to  the  abovementioned  requirements. 


decision  3 

problem-solving  l  i.o.t.  accomplish  work  task 

supervision  J  Environmental  Conditions 

Resources 


perform  certain  sub-tasks 


to  be  designed 


Task 


Artificial  Cognitive  Unit  Operation-Assisting  Means 

(e.g.  Autopilot) 


Work  Object  ..{e.g.  flying  a  combat  A/C) 


can  be  adapted 


Result 


externally  given 

Figure  5-13:  Artificial  Cognitive  Unit  (ACU)  Mimicking  Human  Operator  in  a  “Work  System”. 


Strictly,  Figure  5-13  is  not  representing  a  work  system  any  longer,  since  the  presence  of  a  human  operator  as 
part  of  the  operating  element  is  required  by  definition  [25].  Such  a  system  would  be  degenerated  from  the 
standpoint  of  “work”,  just  existing  for  its  own  sake  and  not  serving  any  human  purpose.  As  soon  as  the  human 
is  involved  as  the  tasking  and  monitoring  element,  which  is  always  the  case,  the  human  will  be  part  of  the 
work  system.  The  implications  of  this  statement  shall  be  discussed  in  the  subsequent  paragraph. 


5.3.2.2  Automatic  and  Autonomous  Performance 

The  (theoretical)  configuration  depicted  in  Figure  5-13,  where  the  system  is  functioning  (i.e.,  transforming  the 
work  object,  e.g.,  flight,  according  to  a  work  task  into  a  desired  work  result)  independently  from  any  human 
intervention,  can  be  referred  to  as  being  an  autonomous  system  with  respect  to  that  particular  work  task. 
For  this  definition  of  autonomy  a  crucial  factor  is  that  a  full  work  system  is  considered.  Automated  part-tasks, 
such  as  autopilot  functions,  working  independently  from  human  intervention  likewise,  are  considered  to  be 
automatic. 

Figure  5-14  [10]  has  to  be  understood  in  connection  with  the  Figures  5-6  and  5-7.  It  shows  the  separation  of 
automatic  and  autonomous  systems  from  a  more  general  point  of  view.  The  framed  elements  in  Figure  5-14 
form  the  considered  work  system. 
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Operator  / 
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Operator  / 
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Mission  Objective 


Figure  5-14:  Comparison  between  Automatic  and  Autonomous  Mission 
Accomplishment  (Framed  Elements  Form  Work  System). 


In  case  A  of  Figure  5-14  the  work  system  is  consisting  of  a  human  operator  (pilot)  and  the  vehicle,  the  latter 
representing  the  work  object  and  the  operation-assisting  means,  i.e.,  the  conventional  setup  of  a  manned 
vehicle.  In  this  configuration  the  operation-assisting  means  will  provide  diverse  automatic  functions.  Having 
conventional  manned  vehicles  or  aircraft,  an  external  command  and  control  unit  works  out  a  mission  order  as 
a  representation  of  the  desired  mission  objective  and  passes  it  to  the  operator,  who  accomplishes  the  mission. 
Such  a  work  system  acts  autonomously  and  co-operatively,  depending  on  the  current  situation,  the  goals, 
the  system’s  and  operator’s  capabilities  and  resources. 

Case  B  of  Figure  5-14  represents  the  solution  of  conventional  automation  to  the  guidance  of  an  uninhabited 
vehicle  (compare  Figure  5-7).  In  this  setup  the  work  system  is  spatially  dislocated,  bearing  the  aforementioned 
restraining  factors  for  remote  operation.  The  vehicle  itself  may  be  considered  as  being  semi-automatic  in  the 
case  of  loose  supervision  or  even  fully  automatic  if  no  monitoring  or  supervision  is  desired  at  all.  Such  a 
vehicle  typically  has  no  4 on-board  intelligence’,  and  therefore,  will  accomplish  a  mission  automatically. 
In  some  occasions,  if  there  is  a  person  on  ground  acting  as  a  remote  operator  within  the  guidance  loop,  he  has 
some  influence  on  the  actions  of  the  vehicle  during  operation  and  the  vehicle  acts  partially  automatically. 
Otherwise,  the  person  takes  more  the  role  of  a  supervisor,  who  usually  provides  the  vehicle  with  pre-planned 
instructions,  possibly  including  some  action  alternatives.  How  adequate  an  automatic  vehicle  reacts  to  a 
situation  change  depends  in  case  of  operator-guided  operation  on  whether  the  operator  gets  enough 
information  about  the  situation  in  which  the  vehicle  is  located.  If  a  vehicle  operates  fully  automatically,  it  can 
only  react  to  situation  changes,  which  were  foreseen  by  the  operator. 

Case  C  of  Figure  5-14  is  the  situation  where  an  autonomously  performing  work  system  is  only  consisting  of 
machine  elements,  i.e.,  the  vehicle  including  operation-assisting  means  and  the  ACU,  which  is  capable  of 
generating  human-like  behaviour.  Exclusively  in  this  configuration  the  remote  agent  (i.e.,  the  vehicle  and  its 
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guidance)  forms  an  autonomous  entity  itself.  Having  this  capability  on-board  several  vehicles  with  partially 
overlapping  (i.e.,  partly  equal  and  partly  different)  resources  and  capabilities,  it  becomes  possible  to  have  a 
mission  accomplished  autonomously  and  co-operatively  with  an  external  supervisor  providing  an  overall 
mission  objective  to  all  of  them.  [10] 

5.3.2.3  Cognitive  Automation  as  Part  of  the  Operation-Assisting  Means 

In  a  traditional  sense  the  human  operator  provides  capability  of  cognition  within  a  conventional  work  system, 
whereas  the  operation-assisting  means  do  not.  As  an  alternative  to  full  autonomy  without  human  intervention 
a  configuration,  where  an  artificial  cognitive  component  in  addition  to  the  human  operator  might  be 
introduced  into  the  work  system. 

Figure  5-15  [24]  shows  the  ACU  being  part  of  the  operation-assisting  means  in  an  otherwise  conventional, 
manned  work  system  setup. 


Environmental  Conditions 
Resources 

i 


Human  Operator 


Task 

. ft  * . 

Result 

Work  Object  (e.g.  flying  a  combat  A/C) 


Figure  5-15:  Work  System  with  ACU  in  Configuration  “Cognitive 
Automation  as  Part  of  Operation-Assisting  Means”. 


“As  opposed  to  conventional  automation ,  cognitive  automation  works  on  the  basis  of 
comprehensive  knowledge  about  the  work  process  objectives  and  goals  [...],  pertinent  task 
options  and  necessary  data  describing  the  current  situation  in  the  work  process.  [...]  Making  use 
of  these  capabilities  in  terms  of  operation-assisting  means  in  the  work  system ,  it  has  no  longer  to 
be  the  exclusive  task  of  the  [human]  operator  to  monitor  the  process  subject  to  the  prime  work 
system  objectives.  ”  [24] 

In  the  case  of  cognitive  automation  incorporated  into  the  operation  assisting  means  the  vision  of  a  “cognitive 
autopilot”,  as  opposed  to  the  conventional  autopilot  mentioned  earlier  on,  would  certainly  perform  superiorly. 
Once  activated,  a  “cognitive  altitude  acquire”  function  would  check  the  mode  selection  and  the  demand 
setting  against  the  context  of  the  current  mission  task.  It  would  notice  ground  or  traffic  proximity,  or  maybe 
exposure  to  enemy  radar.  It  would  conclude  that  these  conditions  will  result  in  loss  of  the  mission  or  even 
disaster.  Finally,  it  would  work  out  an  appropriate  solution,  either  by  indicating  to  the  human  operator  the 
disturbance  or  by  suggesting  or  even  performing  corrective  actions. 
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This  is  pretty  much  the  basic  idea  of  a  cognitive  assistant  system.  Several  research  activities  proved  this  concept 
more  or  less  recently,  the  Cockpit  Assistant  System  CASSY  [11],  the  Crew  Assistant  Military  Aircraft  CAM  A 
[12,13],  and  the  Tactical  Information  and  Mission  Management  System  TIMMS  [14].  Some  more  information 
on  these  projects  will  be  given  at  the  end  of  this  chapter.  Onlcen  summarises  the  requirements  for  this  class  of 
systems: 

“(1)  It  must  be  ensured  the  representation  of  the  full  picture  of  the  flight  situation,  including  that 
the  attention  of  the  cockpit  crew  is  guided  towards  the  objectively  most  urgent  task  or  sub-task  as 
demanded  in  that  situation. 

(2)  A  situation  with  overcharge  of  the  cockpit  crew  might  come  up  even  when  situation 
awareness  has  been  achieved  by  the  pilot  crew.  In  this  case  the  assistant  system  has  to  transfer 
the  situation  into  a  normal  one  which  can  be  handled  by  the  crew  in  a  normal  manner.  ”  [15] 

In  these  so-called  two  basic  requirements  for  human-machine  interaction  the  way  is  paved  already  for  the  next 
step  in  the  integration  of  cognitive  automation  in  a  work  system,  in  the  sense  of  cognitively  facilitated  human- 
machine  co-operation  as  another  alternative  work  system  configuration. 


5.3.2.4  Co-operative  Automation  as  By-Product  of  Cognitive  Automation 

As  opposed  to  mere  interaction,  co-operation  has  particular  characteristics.  Co-operating  units  in  a  work  system 
pursue  additional  goals.  Billings  [2]  formulates  respective  design  principles  for  human-machine  co-operation  in 
the  context  of  human  centred  design: 

The  human  operator  must  be 

•  Actively  involved; 

•  Adequately  informed;  and 

•  Able  to  monitor  the  automation  assisting  him. 

The  automated  systems  must 

•  Be  predictable;  and 

•  Also  be  enabled  to  monitor  the  human  operator. 


And, 


•  Every  intelligent  system  element  must  know  the  intent  of  other  intelligent  system  elements. 

Billings,  though,  does  not  offer  a  solution  how  the  intelligent  machine  elements  shall  be  designed,  yet. 
He  does  not  bear  machine-machine  co-operation  in  mind,  either. 

Figure  5-16  [24]  shows  a  work  system  setup,  where  the  human  operator  and  the  ACU  form  a  team.  In  this 
configuration  the  ACU  has  reached 

“[...]  the  high-end  authority  level  for  decisions  in  the  work  system,  which  was,  so  far,  occupied 
by  the  human  operator  alone.  ”  [24] 
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Environmental  Conditions 
Resources 


Figure  5-16:  Work  System  with  ACU  in  Configuration  “Co-operative  Automation”. 

As  a  consequence  of  this  consideration  each  of  the  team  members  has  to  have  the  ability  to  carry  out  all  tasks, 
which  might  be  crucial  for  the  performance  of  the  overall  work  task.  A  crew  co-ordination  concept, 
very  similar  to  one  examined  for  human-human  cockpit  teams  [3,16],  has  to  be  developed. 

5.3.3  Technological  Challenges 

In  the  previous  section  the  introduction  of  artificial  cognitive  capabilities  in  a  work  system  was  discussed. 
The  perspective  of  this  advanced  automation  technology  approach  is  to  overcome  current  problems  with 
clumsy  systems  in  human-machine  co-operation,  in  order  to  facilitate  machine  autonomy  without  human 
intervention,  and  to  support  human  operators  in  demanding  tasks,  which  tend  to  overload  human  resources. 
The  term  of  an  Artificial  Cognitive  Unit  (ACU)  has  been  introduced,  so  far  without  explaining,  how  such  a 
system  element  shall  be  constructed. 

Figure  5-17  visualises  the  various  technological  challenges  to  be  borne  in  order  to  implement  such  a  system: 

•  Comprehensive  situation  perception:  Figure  5-5  shows  that  in  principle  the  human  operator  on-board 
has  access  to  information  (environmental  stimuli)  which  is  offered  to  him  in  addition  to  the  information 
from  his  vehicle  systems.  The  human  operator  is  able  to  look  out  of  the  window  of  his  vehicle;  he  can 
hear  environmental  noise  or  follow  the  voice  communication  on  the  radio.  He  can  sense  structural 
vibrations  of  his  vehicle  and  even  smell  smoke  in  the  cabin,  however,  the  most  important  point  is 
probably  that  the  human  operator  has  the  principal  capability  to  understand  most  of  these  perceptions 
and  put  them  into  the  context  of  previous  experiences.  Conventional  automation  is  lacking  most  of  these 
abilities,  and  thereby,  has  no  access  to  a  wide  spectrum  of  environmental  information  relevant  for 
crucial  decisions.  In  order  to  facilitate  cognitive  behaviour  in  a  machine  system  the  ability  to  perceive 
the  environment  has  to  be  ensured.  Dickmanns  and  his  research  group  contributed  very  substantial  work 
in  the  area  of  computer  vision  for  autonomous  road  vehicle  guidance  (e.g.,  [17]). 

•  Cognitive  capabilities:  The  next  step  after  a  successful  perception  of  the  world  will  be  the  deduction 
of  rational  behaviour  on  the  basis  of  the  gathered  information.  Therefore,  further  cognitive 
capabilities  (of  course,  perception  is  a  cognitive  capability  itself  already)  will  be  needed,  both,  on  the 
human  operator’s  side  as  well  as  on  behalf  of  the  machine.  What  humans  can  do  seemingly 
effortlessly  has  to  be  given  to  the  automation  by  design.  Automation  shall  be  enabled  to  built  up  a 
mental  model  of  the  surrounding  world,  which  can  be  understood  as  the  comprehension  of  the 
situation  and  its  projection  into  the  future  (e.g.,  [18]  as  one  point  of  view).  The  so-gained  situational 
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knowledge  shall  be  adequately  represented  in  memory  (e.g.,  [19]  or  [20]  as  two  classical  sources). 
On  the  basis  of  this  situation  specific  knowledge  and  other  pre-recorded  knowledge,  problem-solving 
and  decision-making  shall  be  performed  in  order  to  achieve  certain  goals.  The  modelling  of  this 
component  of  cognition  will  be  the  main  subject  of  the  next  section  of  this  chapter,  resulting  in  the 
theory  of  the  Cognitive  Process  [24]. 

•  Human-machine  interaction:  Having  an  intelligent  unit  within  the  work  system,  which  is  enabled  to 
gather  and  understand  the  entire  situation,  to  make  decisions  and  to  exhibit  rational  and  goal-oriented 
behaviour,  it  will  be  necessary  to  make  it  interact  with  the  human  operator.  First  of  all,  appropriate 
communication  channels  have  to  be  found.  A  system  designed  to  perform  on  the  higher  levels  of 
cognition  certainly  offers  the  principal  opportunity  to  use  language  as  communication  code  [21], 
besides  others.  Furthermore,  an  appropriate  co-ordination  technique  has  to  be  found  in  order  to 
facilitate  a  fruitful  co-operation  aiming  upon  the  accomplishment  of  a  common  mission  objective. 
In  the  long  term,  intelligent  machines  shall  appreciate  other  intelligent  agents  in  their  environment, 
either  human  or  artificial,  as  such.  In  this  case,  co-operation  will  be  an  additional  behaviour  of  a 
machine,  based  upon  cognitive  capabilities  [10]. 

•  Level  of  automation  and  authority:  Like  with  human  teams,  the  question  of  the  allocation  of  tasks  and 
authorities  has  to  be  answered  for  human-machine  teams,  as  well  as  for  machine-machine  teams. 
Weiner  [3]  investigated  the  issue  of  crew  resource  management  for  the  aviation  domain.  Billings  [2] 
made  his  suggestions  for  human  centred  aircraft  automation  design  and  human-machine  co-operation. 
Taylor  [22]  worked  on  the  problem  of  allocation  of  authorities  within  a  human-machine  team  with  the 
aim  to  provide  the  necessary  and  sufficient  levels  of  authority  for  the  task  automation  -  but  still, 
only  the  existence  of  artificial  cognitive  team  mates  will  reveal  the  critical  questions  in  the  context  of 
the  allocation  of  tasks  and  authority,  which  have  to  be  tackled. 

•  Paradigm  shift:  Finally,  users,  consumers,  designers,  companies,  procurement  officers,  and  customers, 
who  are  involved  in  the  introduction  of  a  new  automation  technology  in  their  specific  way,  have  to 
reconsider  the  issue  of  the  evolution  of  personal,  social,  and  economic  factors,  which  comes  along  with 
such  a  process.  In  some  cases  a  paradigm  shift  will  be  inevitable.  At  the  very  deep  end  of  the  chain, 
training  issues  related  with  the  handling  of  somehow  intelligent  machinery  will  certainly  come  up. 


Figure  5-17:  Technological  Challenges  in  Advanced  Automation. 
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The  following  section  “Approaching  Cognition”  will  be  dedicated  to  the  analysis  of  cognitive  capabilities  of 

humans.  On  the  basis  of  this,  an  overview  over  information  technology  approaches  to  artificial  cognition  will 

be  given.  As  a  result,  the  so-called  Cognitive  Process  (CP)  as  a  theoretical  approach  to  machine  intelligence 

will  be  introduced. 
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5.4  APPROACHING  COGNITION 

In  the  previous  sections  the  term  “cognition”  has  been  used  rather  sloppy  in  the  sense  of  a  particular  human 
capability,  and,  hopefully,  of  a  future  machine  function.  This  section  shall  sort  things  out  in  terms  of  how 
humans  perform  and  how  a  machine  has  to  be  constructed  in  order  to  exhibit  intelligent  behaviour,  likewise. 
“Intelligence”,  a  term  which  can  be  replaced  by  “cognition”  in  most  cases  in  this  context,  is  defined  rather 
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vaguely  defined  in  habitual  language  use,  although  being  a  rather  valid  concept  in  psychology.  Besides  many 
other  definitions,  Morris  [1]  gives: 

“Intelligence  [...  is  ...]  a  general  term  encompassing  various  mental  abilities ,  including  the 
ability  to  remember  and  use  what  one  has  learned,  in  order  to  solve  problems,  adapt  to  new 
situations,  and  understand  and  manipulate  one's  environment.  ”  [1] 

Nowadays  intelligence  or  cognition  is  no  longer  exclusively  considered  by  psychology,  but  is  subject  to  the 
interdisciplinary  field  of  “cognitive  science”,  which  is  influenced  by  philosophy,  psychology,  neuroscience, 
linguistics,  anthropology,  and,  of  course,  by  computer  science  and  information  technology.  In  this 
enumeration  the  last  discipline  seems  to  be  of  some  particular  interest,  because  it  facilitates  to  prove  the 
validity  of  theories  by  modelling  and  simulation.  New  concepts  emerging  in  the  field  of  Artificial  Intelligence 
(AI),  a  field  of  computer  science  that  attempts  to  develop  intelligently  behaving  machines  [Anderson,  2000], 
influenced  the  cognitive  psychology,  and  vice  versa  [2]. 

Many  approaches  dedicate  themselves  to  the  exploration  of  the  underlying  processing  structure,  as  opposed  to 
the  principles  of  behaviourism,  which  was  a  rather  strong  trend  in  the  early  20th  century  psychology, 
only  being  concerned  with  the  externally  observable  behaviour  of  a  human. 

Figure  5-18  depicts  the  different  approaches,  the  one  of  the  behaviourism  (top),  and  the  alternative  modelling 
view  considering  the  internal  processing  (bottom).  In  both  cases,  the  information  processing  paradigm 
(input  ->  processing  ->  output)  is  appropriate  to  characterise  the  phenotype  of  the  situation.  The  behaviourism 
searches  for  the  input-output  mapping  of  human  behaviour,  no  matter  how  it  will  be  implemented. 
Other  modelling  approaches  focus  on  the  description  of  the  underlying  processes,  in  order  to  expose  the 
observed  behaviour.  In  the  subsequent  few  sections  a  brief  overview  will  be  given  over  approaches  to  the 
modelling  of  the  processing  mechanisms  of  human  cognition.  Behaviour,  in  turn,  can  be  utilised  in  order  to 
validate  related  models. 


Figure  5-18:  Modelling  Behaviour  or  Processing. 


5.4.1  Model  of  Human  Performance 

In  order  to  open  the  window  to  the  development  of  human-like  performance  features  in  terms  of  cognitive 
automation,  the  very  well  accepted  model  of  human  performance  levels  by  Jens  Rasmussen  [3]  will  be 
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consulted,  to  begin  with.  The  simplicity  and  intelligibility  made  this  model,  originally  having  its  seeds  in 
ergonomics  research,  quite  popular  in  the  circles  of  cognitive  psychologists  as  well  as  amongst  engineers. 
In  fact,  Rasmussen’s  model  became  the  probably  most  common  psychological  scheme  within  the  entire 
engineering  community. 

Without  going  into  too  much  detail  here  (for  a  most  detailed  discussion  refer  to  [3]  and  [26]),  the  model 
distinguishes  between  three  levels  of  human  performance,  the  skill-based,  the  rule-based,  and  the  knowledge- 
based  behaviour  (see  Figure  5-19).  On  the  skill-based  level  highly  automated  control  tasks  will  be  performed, 
without  any  mental  effort  or  consciousness.  Typical  for  this  level  is  the  continuous  control  of  the  body  in 
three-dimensional  space  and  time.  Most  of  this  performance  is  carried  out  in  feedforward  control  mode  by 
pre-programming  of  stored  sensor-motor  patterns  on  the  basis  of  task  specific  features.  Typical  behaviour  on 
this  level,  like  tracking  a  road,  will  be  assembled  by  running  a  sequence  of  parameterised  templates  with  some 
feedback  control  ratio  for  precision  enhancement. 


Figure  5-19:  Rasmussen’s  Model  of  Human  Operator’s  Performance  Levels  Linked  to  Environment. 

On  the  rule-based  level  most  of  the  everyday  conscious  action  that  we  perform  takes  place  in  a  strict 
feedforward  control  manner.  Here,  humans  follow  pre-recorded  scripts  and  procedures  in  order  to  activate  the 
appropriate  sensori-motor  patterns  on  the  basis  of  the  presence  of  clearly  recognised  objects  characterising  the 
prevailing  situation.  With  training  formerly  rule-based  performance  tends  to  be  dropped  to  the  skill-based 
level.  Rule-based  performance  is  goal-oriented,  although  goals  are  not  explicit,  but  encoded  in  the  pre¬ 
conditions  of  the  applicable  rules. 

The  knowledge-based  level  will  be  entered  in  situations,  where  there  are  no  applicable  rules  available  in  order 
to  recognise  objects  or  to  determine  the  selection  of  action.  This  is  the  case  when  the  situation  requires  the 


5-24 


RTO-TR-HFM-078 


dMNATO 
WP  I  OTAN 


ARTIFICIAL  COGNITION  AND  CO-OPERATIVE  AUTOMATION 


preoccupation  with  a  non-pre-defined  problem.  In  this  case  general  concepts  have  to  be  consulted  in  order  to 
identify  the  situation,  i.e.,  find  similar  or  somehow  related  situations  in  previous  experience.  Goals  derived 
from  overall  aims  explicitly  direct  the  tasking.  Planning,  i.e.,  problem-solving  will  be  deployed  in  order  to 
generate  new  scripts  or  procedures,  which  will  be  executed  on  the  rule-based  level.  In  general,  problem¬ 
solving  can  be  considered  as  a  highly  versatile  process,  incorporating  strategies  such  as  difference  reduction 
and  means-ends  analysis  [17]  as  well  as  search  in  problem  space  [4].  So-called  mental  models  will  be  the 
knowledge  basis  for  the  highest  performance  level  [3]. 

Although  it  is  so  common  to  the  engineering  community,  because  of  its  apparent  use  of  clear  functional 
blocks  and  their  interrelations,  Rasmussen’s  model  deserves  some  interpretation  from  an  information 
technology  point  of  view.  One  reason  for  this  is  the  improper  handling  of  knowledge  in  the  model.  Most  of  the 
boxes  represent  a  dedicated  function  or  processing  step  (e.g.,  ‘recognition’,  ‘planning’).  Only  two  particular 
boxes  (i.e.,  ‘stored  rules  for  tasks’  and  ‘sensori-motor  patterns’)  represent  knowledge,  without  having  their 
individual  functions  specified.  And  finally,  only  one  functional  block  (i.e.,  ‘decision  of  task’)  makes  use  of  an 
explicit  knowledge  basis  (‘goals’).  From  an  information  technology  standpoint  it  would  be  desirable  to  modify 
the  model  according  to  the  following  guidelines,  at  least  for  a  first  step  of  advancement: 

•  Use  boxes  for  functions  or  processing  steps; 

•  Label  the  knowledge  which  is  made  use  of  in  each  box;  and 

•  Label  all  inputs  and  outputs  of  the  functional  blocks. 

The  detailed  discussion  of  this  issue  shall  be  the  matter  of  forthcoming  publications. 

5.4.2  Modelling  Approaches  for  Intelligent  Machine  Behaviour 

As  discussed  above,  the  human  performance  can  be  decomposed  in  several  high  level  cognitive  functions, 
which  rely  upon  certain  a-priori  knowledge.  Besides  the  task-related  a-priori  knowledge,  there  are 
mechanisms  necessary  in  order  to  process  this  knowledge.  Highly  related  with  these  mechanisms  is  the  form 
of  representation  of  this  knowledge.  In  parallel  to  the  development  of  psychological  performance  and 
behaviour  models  as  briefly  discussed  in  the  previous  section  there  takes  place  the  development  of 
technological  approaches  to  intelligent  machine  behaviour,  each  of  which  influencing  and  fertilising  one 
another. 

From  a  very  global  standpoint  there  can  be  identified  two  fundamentally  different  approaches,  one  strongly 
influenced  by  the  idea  of  mimicking  the  human  implementation  of  cognition  in  the  brain  (i.e.,  connectionism, 
artificial  neural  networks,  sub-symbolic  AI)  [5],  and  the  other  being  based  upon  models  taken  from 
information  technology  (i.e.,  symbolism,  Artificial  Intelligence)  [2]. 

Besides  those  two  main  streams,  early  human  factors  research  offered  modelling  approaches  on  the  basis  of 
control  theory  [18]. 

Figure  5-20  shows  the  principal  approach  of  this  class  of  approaches,  modelling  human  behaviour  by  means 
of  transfer  functions.  Typically,  there  were  made  a  couple  of  structural  assumptions,  such  as  reaction  time  and 
neuromotor  delay  as  inherent  parameters  and  gain  and  anticipation  as  task-adaptable  parameters.  On  the  basis 
of  such  model  structure  quite  successful  parameter  identifications  could  be  performed,  typically  limited  to 
various  sensori-motor  control  tasks. 
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Figure  5-20:  Model  of  Human  Behaviour  Motivated  by  Control  Theory  (i.e.,  Transfer  Function). 


Coming  back  to  the  aforementioned  antithetic  approaches  of  connectionism  and  symbolism  one  major 
difference  can  be  identified  in  the  way  of  knowledge  representation.  In  the  connectionism  there  is  no 
separation  existing  between  knowledge  and  its  processing.  Neither  is  knowledge  in  any  way  explicit, 
but  spread  over  the  weights  of  the  connections  between  simple  but  numerous  processing  units  (neurons). 
Each  single  weight  provides  a  contribution  to  the  knowledge  persistent  to  the  model  without  a  particular 
allocation  of  meaning.  The  entirety  of  weights  represents  the  entirety  of  a-priory  knowledge.  Many  models 
provide  learning  mechanisms,  either  in  supervised  or  unsupervised  learning  mode. 

Symbolism,  on  the  other  hand,  utilises  explicit,  meaningful  symbols  in  order  to  handle  knowledge.  Processing 
architectures  are  derived  from  simple  information  processing  paradigms,  as  depicted  in  Figure  5-21. 


Figure  5-21:  Model  of  Human  Processing  Motivated  by  Information  Technology. 


While  the  processor  is  almost  independent  from  the  task,  the  functionality  is  encoded  in  the  knowledge 
persistent  to  the  memory.  The  interface  to  the  external  world  build  dedicated  receptors  and  effectors. 
The  probably  most  famous,  classical  model  of  this  kind  is  the  so-called  CMN-model  [6]  as  shown  in 
Figure  5-22. 


Figure  5-22:  The  Model  Human  Processor  Adapted  from  CMN-Model. 
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The  CMN-model  in  particular  points  out  the  assumed  structure  of  the  memory  of  the  human  and  some 
performance  features  and  limitations  of  its  building  blocks.  The  7-chunk  capacity  limit  of  the  working 
memory  is  probably  one  of  the  most  acquainted  proposition  in  this  context.  As  principal  concept  of  processing 
the  so-called  recognise-act  cycle  (RAC)  (see  Figure  5-23)  is  proposed. 


Figure  5-23:  The  Recognise-Act  Cycle. 

To  characterise  the  activity  of  the  cognitive  processor,  [6]  state: 

On  each  cycle,  the  contents  of  Working  Memory  initiate  associatively-linked  actions  in  the  Long- 
Term  Memory  (“recognize”),  which  in  turn  modify  the  contents  of  Working  Memory  (“act”), 
setting  the  stage  of  the  next  cycle.  [6] 

The  interface  to  the  environment  is  through  the  working  memory. 

The  so-called  production  systems  (expert  systems)  predominantly  follow  the  processing  approach  of  the 
recognise-act  cycle  using  mainly  IF-THEN  rules  as  knowledge  representation  form  for  heuristics  and  “rules  of 
thumb”.  Figure  5-24  shows  the  main  building  blocks  of  such  a  rule-based  system  (i.e.,  production  system). 
The  knowledge  is  stored  in  the  rule  base,  the  long-term  memory  of  the  architecture.  Based  upon  the  short-term 
(i.e.,  working)  memory  contents  (i.e.,  internal  states  plus  input  from  and  output  to  the  environment)  according 
to  their  pre-conditions  rules  from  the  rule  base  will  be  selected  as  candidates  for  execution.  After  the  solution 
of  conflicts  (in  the  case  of,  e.g.,  more  than  on  applicable  rules)  the  rule  will  be  “fired”,  i.e.,  the  post-condition 
of  the  rule  will  be  executed  in  order  to  modify  the  content  of  the  short-term  memory,  either  initiating  a 
succeeding  recognise-act  cycle  base  on  internal  state  changes,  or  evoking  an  action  at  the  output. 


Figure  5-24:  Architecture  of  a  Rule-Based  System  (i.e.,  Production  System). 
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Besides  these  very  traditional  approaches,  predominantly  relying  on  the  use  of  rules  as  form  of  knowledge 
representation,  many  other  kinds  of  knowledge  representations  evolved  in  the  era  of  GOFAI  (“Good-Old- 
Fashioned  Artificial  Intelligence”),  most  of  which  linked  to  symbolist  approaches  on  one  or  the  other  way, 
e.g.,  semantic  networks  [19],  conceptual  dependency  [7],  frames/schemata  [20],  scripts  [8],  just  to  name  the 
classical  ones. 

Besides  these  “classical”  ones  there  are  at  least  two  more  recent  approaches  important  to  be  mentioned  here, 
both  of  which  being  symbolic  cognitive  architectures  meant  to  model  intelligent  performance: 

•  ACT-R  [9]  is  used  to  model  different  aspects  of  human  cognitive  behaviour,  i.e.,  to  implement 
human-like  behaviour.  ACT-R  has  its  starting  point  in  creating  a  computational  theory  of  human 
memory.  It  combines  predominantly  symbolic  representations  with  sub-symbolic  mechanisms, 
mainly  to  model  human  performance  aspects  such  as  the  limited  retrievability  of  knowledge. 

•  SOAR  [10,4]  is  used  to  model  an  agent’s  intelligent  capabilities,  i.e.,  to  implement  rational  behaviour. 
SOAR  has  its  roots  in  the  attempt  to  understand  the  methodological  and  structural  pre-requisites  of 
human  problem-solving  and  decision-making.  Concerning  knowledge  representations,  SOAR  is  a 
rule-based,  i.e.,  a  production  system. 

While  the  aforementioned  architectures  pair  a  still  strong  focus  on  knowledge  representation  with  architectural 
aspects  of  cognition,  some  concurrent  approaches  capitalise  upon  mostly  architectural  views.  Some  of  the 
most  prominent  approaches  shall  be  brought  up  here: 

•  BDI  (Belief-Desire-Intent)- Agents  [11]:  Agents  are  software  constructs  situated  in  a  certain 
environment  and  interacting  with  it  autonomously  in  order  to  achieve  specific  individual  objectives. 
Applications  are  widely  spread  over  various  domains  from  data  management  over  user  interfaces  and 
computer  mediated  collaboration  to  robotics.  The  BDI  architecture  suggests  the  usage  of  mental 
attitudes  representing  the  informational  (belief),  the  motivational  (desire)  and  the  deliberative  (intent) 
state  of  the  agent. 

•  RCS  (Real-time  Control  System)  [12]:  RCS  is  a  reference  model  architecture,  suitable  for  real-time 
control  problem  domains,  and  therefore  closely  related  to  robotics.  It  focuses  on  intelligent  control 
that  adapts  to  uncertain  and  unstructured  operating  environments.  The  architecture  provides  a 
top-down  hierarchical  composition  of  processing  nodes  incorporating  the  cognitive  functions  of 
sensory  processing,  world  modelling,  value  judgement  and  behaviour  generation. 

•  Subsumption  Architecture  [13],  representing  the  field  of  behaviour-based  robotics,  almost  fully 
dismisses  the  notion  of  a  mental  world  model.  Instead,  this  architecture  is  strongly  behaviour 
oriented,  i.e.,  focussing  on  direct  perception-action  mappings  facilitated  by  close  couplings  between 
sensors  and  actuators.  More  complex  behaviours  are  assumed  to  emerge  from  simpler  ones  in  a 
bottom-up  manner.  Symbolic  representations  are  not  part  of  this  architecture. 

As  this  very  brief,  and  by  no  means  complete,  overview  of  modelling  approaches  for  intelligent  machine 
behaviour  indicates,  the  research  focus  over  the  last  three  decades  has  been  shifted  from  mostly  method 
oriented  approaches,  e.g.,  how  to  represent  knowledge,  to  somewhat  more  architecture  focussed  approaches. 

When  it  shall  come  down  to  a  systems  engineering  implementation  of  intelligent  machinery,  both  aspects 
yet  are  of  their  particular  importance,  and  therefore  should  be  considered  in  a  well  balanced  manner. 
The  following  sub-sections  introduce  the  concept  of  the  Cognitive  Process  [21,22,14],  which  comprises  a 
theory  based  on  cognitive  psychology  with  a  knowledge-based  architecture. 
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5.4.3  The  Cognitive  Process  as  Approach  to  Cognitive  Automation 

Coming  back  to  the  aim  of  a  co-operative  structure  of  a  human-machine  system  (as  depicted  in  Figure  5-12), 
the  notion  of  cognitive  automation  (as  introduced  in  the  Sections  5.3.2.  ff),  and  the  technological  challenge  of 
providing  cognitive  capabilities  to  an  Artificial  Cognitive  Unit  (ACU)  (as  formulated  in  Section  5.3.3), 
we  now  want  to  take  the  findings  on  cognition  (Section  5.4)  into  consideration  in  order  to  develop  a  theory- 
based  architecture  for  intelligent  machine  behaviour.  Findings  from  cognitive  psychology  and  artificial 
intelligence  shall  be  taken  into  consideration  likewise. 

The  concept  of  a  piece  of  automation  being  a  team-player  in  a  mixed  human-machine  team,  or  even  a  machine 
taking  over  responsibility  for  work  objectives  to  a  large  extent,  promotes  the  approach  of  deriving  required 
machine  functions  from  models  of  human  performance.  In  Section  5.4.1  Rasmussen’s  model  has  been 
introduced. 

When  we  look  at  conventional  automation  as  discussed  in  the  Section  5.2.2  and  5.3.1,  in  particular  in  the 
avionics  domain,  it  mainly  acts  on  a  level  which  might  be  compared  with  the  skill-based  human  performance 
level  (e.g.,  flight  control  systems,  autopilot  systems).  Some  functionalities  might  be  attributed  to  the  rule- 
based  (e.g.,  traffic  collision  avoidance  systems)  and  few  on  the  knowledge-based  level  (e.g.,  mission  planning 
support  in  flight  management  systems). 

On  the  other  hand,  not  many  automation  systems  can  be  identified,  providing  an  understanding  of  the  current 
situation  in  terms  of  recognition  and  identification,  or  considering  goals,  which  are  essential  for  the  decision 
of  what  to  do  next  in  an  unknown  situation,  as  already  discussed  in  Section  5.3.1. 

In  contrast  to  the  conventional  approach,  cognitive  automation  aims  for  rationality  in  a  human-like 
performance  manner,  without  modelling  typical  human’s  shortcomings.  Thus,  all  functions  of  Rasmussen’s 
model  have  to  be  covered,  including  those  already  incorporated  in  conventional  automation  [14].  Figure  5-25 
shows  the  main  focus  area  for  future  developments  aiming  at  cognitive  automation,  namely  the 
implementation  of  a  comprehensive  situation  understanding  and  goal-driven  decision-making,  as  high  level 
cognitive  capabilities. 
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situation  understanding  and  goal-driven  decision  not  covered  by  operation-assisting  means 


Figure  5-25:  Conventional  and  Cognitive  Automation  Explained 
by  Rasmussen’s  Model  of  Human  Performance. 


In  order  to  achieve  a  system  engineering  framework,  the  main  idea  of  Rasmussen’s  model,  namely  rule-  and 
knowledge-based  performance,  is  mapped  into  the  so-called  Cognitive  Process  (CP).  The  CP  is  an  approach 
to  modelling  human  information  processing,  which  is  suitable  for  providing  human-like  rationality  [14,15]. 
As  it  is  compatible  with  human  cognition,  and  the  generated  behaviour  is  driven  by  goals,  which  are 
represented  explicitly,  it  is  well  suited  for  the  development  of  a  cognitive  system,  which  is  part  of  a  team 
consisting  of  artificial  and/or  human  team  mates. 

Figure  5-26  shows  the  CP  consisting  of  the  body  (inner  part)  and  the  transformers  (outer  extremities). 
The  body  contains  all  knowledge,  which  is  available  for  the  CP  to  generate  behaviour.  There  are  two  kinds  of 
knowledge:  the  4 a-priori  knowledge’,  which  is  given  to  the  CP  by  the  developer  of  an  application  during  the 
design  process  and  which  specifies  the  behaviour  of  the  CP,  and  the  'situational  knowledge’,  which  is  created 
at  run  time  by  the  CP  itself  by  using  information  from  the  environment  and  the  a-priori  knowledge. 
The  functional  units  effectively  processing  knowledge  are  the  above-mentioned  transformers,  which  read 
input  data  in  mainly  one  area  of  the  situational  knowledge,  use  a-priori  knowledge  to  process  the  input  data, 
and  write  output  data  to  a  designated  area  of  the  situational  knowledge. 
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The  following  steps  are  performed  by  the  CP  in  order  to  generate  behaviour: 

•  Information  about  the  current  state  of  the  environment  (input  data)  is  acquired  via  the  input  interface. 
In  this  context,  the  environment  includes  other  objects  in  the  physical  world,  e.g.,  another  UAV  or  an 
obstacle,  as  well  as  the  underlying  vehicle  of  the  CP.  Therefore,  the  input  data  may  for  instance 
contain  information  about  the  current  autopilot  mode  or  pre-processed  sensor  information. 

•  The  input  data  are  interpreted  to  obtain  an  understanding  of  the  external  world  (belief). 
The  interpretation  uses  environment  models,  which  are  concepts  of  elements  and  relations  that  might 
be  part  of  the  environment,  to  build  this  internal  representation. 

•  Based  on  the  belief,  it  is  determined,  which  of  the  desires  (potential  goals)  are  to  be  pursued  in  the 
current  situation.  These  abstract  desires  are  instantiated  to  active  goals  describing  the  state  of  the 
environment,  which  the  CP  intends  to  achieve. 

•  Planning  determines  the  steps,  i.e.,  situation  changes,  which  are  necessary  to  alter  the  current  state  of 
the  environment  in  a  way  that  the  desired  state  is  achieved.  For  this  planning  step,  models  of  action 
alternatives  of  the  CP  are  used. 

•  Instruction  models  are  then  needed  to  schedule  the  steps  required  to  execute  the  plan,  resulting  in 
instructions. 

•  These  instructions  are  finally  put  into  effect  by  the  appropriate  effectors  of  the  host  vehicle. 
The  resulting  actions  affect  the  environment,  i.e.,  modify  the  physical  world. 

These  functional  units  represent  an  application-independent  inference  mechanism,  which  processes 
application-specific  knowledge.  This  knowledge-based  design  approach  is  of  great  advantage  when 
implementing  the  CP:  The  inference  mechanism  has  to  be  implemented  only  once,  and  can  then  be  used  for 
different  applications. 
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It  is  desirable  to  reuse  not  only  the  inference  mechanism,  but  also  knowledge  in  different  applications.  For  this 
purpose,  a-priori  knowledge  has  to  be  unitised  in  so-called  ‘packages’,  each  of  which  represents  a  certain 
capability.  As  indicated  in  Figure  5-27,  each  package  (depicted  as  horizontal  layer)  implements  a  capability 
which  is  designed  according  to  the  blueprint  of  the  CP.  Several  packages  together  form  the  complete  system. 
They  are  linked  by  dedicated  joints  in  the  a-priori  knowledge  and  by  the  use  of  common  situational 
knowledge.  When  looking  vertically  on  the  packages,  a  uniform  structure  of  the  a-priori  knowledge  and  its 
order  of  usage  in  terms  of  processing  steps  according  to  the  transformers  of  the  CP  can  be  recognised. 

Architectural  View:  Transformators 


Figure  5-27:  Representing  Multiple  Capabilities  on  Basis  of  the  Cognitive  Process. 
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5.5  COGNITIVE  SYSTEMS  ARCHITECTURE  -  REALISATION  ASPECTS 

This  section  is  supposed  to  point  out  a  perspective  of  how  to  implement  an  Artificial  Cognitive  unit  (ACU) 
on  the  basis  of  the  proposed  theory.  Figure  5-28  depicts  what  has  been  achieved  so  far,  as  a  review  of  the 
previous  sections.  The  starting  point  is  the  human  operator  as  operating  element  in  a  work  system.  In  a  first 
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step  we  model  human  performance  in  terms  of  high  level  cognitive  functions.  The  analysis  of  the  typical  work 
share  in  work  systems  reveals  that  there  are  particular  shortcomings  in  terms  of  these  high  level  cognitive 
functions  on  the  machine  side,  namely  in  the  domain  of  situation  understanding  and  goal-driven  behaviour. 
In  order  to  achieve  this  capability  on  the  machine  side  the  Cognitive  Process  is  proposed  as  underlying  theory 
derived  from  useful  findings  in  cognitive  psychology  and  artificial  intelligence  research,  likewise. 


Figure  5-28:  Method  of  Cognitive  Systems’  Development. 


As  a  next  step  of  the  development  of  a  cognitive  system  according  to  the  approach  of  cognitive  automation, 
the  realisation  of  an  Artificial  Cognitive  Unit  (ACU)  has  to  be  accomplished.  In  the  succession  of  this  section 
an  engineering  framework  for  the  development  of  such  an  ACU  will  be  described:  The  Cognitive  System 
Architecture  (COSA). 

COSA  offers  a  framework  to  implement  applications  according  to  the  theory  of  the  Cognitive  Process. 
It  provides  an  inference  mechanism  and  various  means,  which  make  it  possible  for  the  developer  of  an 
application  to  use  concepts  like  ‘belief,  ‘goal’  and  ‘plan’  rather  than  a  programming  paradigm  based  on  a 
functional  decomposition  of  an  application  [1]. 

COSA  is  composed  of  four  building  blocks  (cf.  Figure  5-29). 

•  The  kernel  implements  the  theory  of  the  CP  and  does  not  contain  any  application-dependent 
information.  Its  only  task  is  to  generate  behaviour  from  knowledge.  In  the  current  implementation,  it 
is  based  on  SOAR  [Laird  et  al.,  1987],  which  is  a  general  rule-based  architecture  for  developing 
systems  that  exhibit  intelligent  behaviour,  as  described  earlier  in  this  chapter.  The  CP-Library 
structures  the  situational  knowledge  according  to  the  theory  of  the  Cognitive  Process  and  coordinates 
the  performance  of  the  transformers  (interpretation,  goal  determination,  planning,  and  plan 
realisation,  cf.  Figure  5-26). 

•  The  application  is  formed  by  several  COSA-compliant  application  components,  which  correspond  to 
packages  (see  Figure  5-27).  The  components  provide  the  a-priori  knowledge  and  may  also  contain 
servers  with  interfaces  to  the  environment  or  for  external  calculations. 
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•  The  front  end  provides  tools  for  the  developer  of  an  application,  which  help  him  to  model  the 
knowledge  for  the  application.  One  part  of  the  front  end  is  CPL  (Cognitive  Programming  Language), 
which  provides  a  programming  support  to  the  implementation  of  the  above-mentioned  mental  notions 
‘belief,  ‘goal’,  and  ‘plan’  as  concepts  for  knowledge  modelling.  The  CPL  code,  which  represents  the 
knowledge  on  a  rather  high  abstraction  level,  is  compiled  into  a  code  representation  understood  by  the 
kernel,  i.e.,  SOAR  in  the  current  implementation. 

•  Finally,  the  distribution  layer  is  responsible  for  the  communication  among  the  modules  of  COSA. 
It  ensures  that  components  and  modules  can  run  on  different  computers  in  a  network. 
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Figure  5-29:  COSA  -  Cognitive  System  Architecture. 
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5.6  APPLIED  SYSTEM  APPROACHES 

The  previous  sections  provided  a  rather  general  concept  of  how  to  approach  artificial  cognition  and 
co-operative  automation,  which  represents  the  current  state  of  the  art  achieved  at  the  Institute  of  System 
Dynamics  and  Flight  Mechanics  at  the  Munich  University  of  the  German  Armed  Forces,  at  least  from  a 
theoretical  standpoint.  On  the  practical  side  this  achievement  could  be  attained  during  the  course  of  several 
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experimental  programmes  in  the  field  over  the  recent  15  years.  Most  of  the  works  were  well  documented  by 
multiple  publications  and  internationally  well  recognised,  e.g.,  [6]  reporting  the  results  of  the  flight  trials  of 
the  worlds  first  comprehensive  knowledge-based  assistant  system  for  flight-deck  crews,  CASSY  (Cockpit 
Assistant  System).  In  the  late  nineties  followed  CAMA  (Crew  Assistant  Military  Aircraft)  as  a  prototype 
system  in  the  military  transport  domain,  e.g.,  [7]  discussing  the  system  architecture,  [Schulte  &  Stiitz,  1998] 
reporting  on  the  successful  simulator  validation  of  the  system,  and  [46]  pointing  out  the  successful  flight  tests 
that  followed. 

A  detailed  discussion  of  the  achievements  of  the  working  group  around  Reiner  Onken  would  go  beyond  the 
scope  of  this  chapter.  Instead  of  doing  that,  the  rest  of  this  chapter  shall  be  dedicated  to  a  broadening  of  the 
view  by  bringing  up  three  perspectives  from  other  research  groups.  The  first  contribution  (Sub-section  5.6.1) 
is  provided  by  Mike  Chamberlin  (UK)  dealing  with  the  issue  of  Artificial  Intelligence  from  a  methods’  point 
of  view.  This  treatment  perfectly  supplements  the  findings  of  Section  5.4,  especially  of  Sub-section  5.4.2, 
which  were  pretty  much  focussed  on  a  unitary  approach.  Sub-section  5.6.1  will  give  a  nice  overview  and 
evaluation  of  further  approaches. 

Common  to  all  attempts  to  computational  intelligence  of  whatsoever  kind  is  the  (open)  question  to  the 
acquisition  of  the  necessary  knowledge  and  it’s  adequate  representation.  In  Sub-section  5.6.2  Jack  Edwards 
(Canada)  tackled  the  problem  of  intelligent,  adaptive  help  system  design  mainly  under  the  consideration  a 
knowledge  design  and  engineering  methodology  that  combines  elements  of  the  CommonKADS  and  IDEF 
methods,  Explicit  Models  Design  and  Perceptual  Control  Theory. 

Finally,  Sub-section  5.6.3  by  Mike  Waters  and  Robert  Taylor  (UK)  opens  the  perspective  for  the  domain  of 
unmanned  underwater  vehicles  as  an  application  domain.  Their  model  of  autonomous  decision  making  renews 
the  idea  of  cognitive  automation  from  a  different  perspective,  rounding  out  the  picture. 

5.6.1  Artificial  Intelligence  (AI)  Methods  Perspective 

In  order  for  unmanned  air  vehicles  to  fulfil  their  envisaged  enhanced  roles  in  the  future  integrated  battlespace, 
there  is  an  unprecedented  requirement  for  flexible  and  autonomous  operation,  representing  a  massive  leap  in 
capability  compared  to  that  of  today’s  systems. 

This  section  critically  examines  the  notions  of  automation  and  autonomy  on  UAV  and  UCAV  platforms,  and 
operator  decision  support;  in  the  contexts  of  current  and  envisaged  platforms,  roles,  responsibilities  and 
missions.  The  relative  merits  and  maturity  of  AI  techniques  and  technologies  are  studied  and  their 
applicability  to  providing  elements  of  autonomous  behaviour  in  UAVs  and  UCAVs  are  considered  in  light  of 
these;  illustrating  how  semi-  and  fully-autonomous  UAVs,  and  operator  decision  support,  can  boost 
flexibility,  survivability  and  mission  effectiveness  of  UAVs/UCAVs,  augmenting  and  enhancing  the 
capabilities  of  the  future  warfighter. 


5.6.1. 1  Introduction  to  AI  Methods 

It  has  long  been  recognised,  and  more  recently  demonstrated  through  deployments  in  the  Gulf,  Bosnia  and 
Afghanistan,  that  UAVs  have  significant  potential  to  enhance  a  force’s  ability  to  project  combat  power. 
Their  range,  persistence,  altitude  and  potential  for  cost  and  manpower  savings  make  UAVs  an  ideal  candidate 
for  the  dull,  dirty  and  dangerous  missions  of  the  future;  augmenting,  replacing  or  even  surpassing  the 
capabilities  of  manned  aircraft. 


5-36 


RTO-TR-HFM-078 


dMNATO 
WP  I  OTAN 


ARTIFICIAL  COGNITION  AND  CO-OPERATIVE  AUTOMATION 


Removing  the  human  from  the  aircraft’s  cockpit  enables  more  efficient  and  cost-effective  platform  designs, 
albeit  with  the  trade  off  that  the  operator  is  now  further  separated  from  the  action,  with  a  commensurate 
possibility  for  loss  of  situational  awareness.  With  the  operator’s  role  needing  to  evolve  from  a  piloting  to  a 
supervisory  capacity,  treating  many  UAVs  as  one  system  in  order  to  achieve  mission  objectives,  there  are 
serious  implications  for  cognitive  workload  [1]. 


5.6.1.2  UAV/UCAV  Autonomy  Requirements 

Small-scale  UAVs  such  as  Micro-UAVs  (MAVs),  are  intended  to  be  deployed  by  ground  forces  for  short- 
range  surveillance.  As  such,  MAVs  are  supposed  to  be  man-portable  and  expendable,  hence  a  fully- 
autonomous  architecture  is  not  required.  Much  larger  UCAV/UAV  platforms,  although  originally  intended  to 
be  expendable,  cannot  afford  this  luxury  as  development  and  operating  costs  continue  to  spiral  upwards. 
With  the  human  operator  removed  from  the  vehicle  and  reduced  to  a  supervisory  role,  rather  than  being  in  full 
control,  a  significant  automatic  and  autonomous  element  needs  to  be  present  for  the  UCAV  to  achieve  any 
level  of  survivability  and  capability  at  all  above  that  of  a  cruise  missile. 

For  a  UCAV  platform  to  be  successful,  it  has  to  rely  on  achievable  technologies,  and  interactions  between 
component  technologies  must  be  carefully  managed.  Development  is  concerned  with  trade-offs  between 
platform  properties:  individual  components  are  themselves  sufficiently  advanced  already.  Constructing  the 
airframe  is  relatively  straightforward;  achieving  desired  levels  of  survivability  and  mission  effectiveness  are 
the  major  challenges  [2].  How  processes  are  integrated  with  the  platform  and  infrastructures  (e.g.,  datalinks) 
is  the  key. 

Currently,  UAVs  such  as  X-45A  are  capable  of  taking  off  and  landing  automatically,  and  following  waypoints 
to  a  target.  They  can  relay  surveillance  data  to  a  controller  for  target  acquisition  and  designation,  and  deploy 
ordnance  to  destroy  a  soft  target. 

The  main  types  of  mission  that  are  likely  to  be  undertaken  by  a  UAV  or  UCAV  are  listed  in  Table  5-1  [3]: 


Table  5-1:  Likely  UAV  or  UCAV  Mission  Types 


SEAD  (Suppression  of  Enemy  Air  Defences) 

Attacking  air  defences,  e.g.,  SAM  sites;  plus  escort  and 
sweep  roles 

Strike 

Attacking  pre-defined  targets;  plus  escort  and  sweep  roles 

TCT  (Time  Critical  Targeting) 

Attacking  targets  within  a  narrow  window  of  opportunity; 
may  involve  long  loitering  times 

Maritime  specific 

AEW,  ASW,  AsuW  tasks;  organic  to  naval  vessel  or  task 
group 

ISTAR  (Intelligence,  Surveillance,  Target 
Acquisition  and  Reconnaissance) 

Functional  requirements  including,  e.g.:  Battlefield 
Surveillance  and  Reconnaissance;  Target  Detection, 
Location  and  Identification;  BDI/BDA 

Communications  Relay 

Extend  LOS  communications,  e.g.,  UHF,  Link  16 
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An  autonomous  UAV/UCAV  would  have  to  undertake  some  of  the  following  actions  during  each  sortie: 

•  System  monitoring. 

•  Airmanship  tasks,  e.g.,  fuel  states,  aircraft  safety. 

•  Health  monitoring/diagnostics,  leading  to  mission  re -planning: 

•  Work  within  battlespace  infrastructure,  i.e.,  datalinks,  communications  (manage  these);  and 

•  Fault  and  damage  tolerant  control  of  the  aircraft,  i.e.,  it  has  to  fly  safely  and  predictably,  work 
around  minor  faults,  and  carry  out  appropriate  procedures  in  the  event  of  major  failures. 

•  Takeoff: 

•  Interoperability  with  ATC  commands,  and  normal  aircraft  handling  patterns  and  procedures  is 
vital,  even  more  so  for  UCAV-N  operating  from  an  aircraft  carrier  (even  down  to  the  envisaged 
level  of  the  air  vehicle  responding  to  voice  commands  from  ATC). 

•  Formation: 

•  A  need  to  operate  with  other  manned  and  unmanned  aircraft,  in  normal  airspace,  and  as  part  of  a 
package. 

•  Ingress  /  Routing  to  target: 

•  Route  to  pre-deflned  target; 

•  Comply  with  airspace  regulations;  and 

•  Avoid  known  and  pop-up  threats,  dynamically  altering  flight  or  even  mission  plans  if  necessary 
with  SA  gained  through  maintenance  of  RASP. 

•  Carry  out  mission: 

•  Compliance  with  Air  Tasking  Orders  (ATO),  SPINS; 

•  ROE,  Laws  of  Armed  Conflict; 

•  Detect,  Identify  and  Acquire  Targets-  a  real-time  process  for  TCT; 

•  Weapons  release  (SEAD,  Strike,  TCT);  and 

•  Battle  Damage  Investigation  /  Assessment. 

•  Egress: 

•  As  ingress;  and 

•  Added  complications  for  Strike,  SEAD,  i.e.,  package  re-formation  and  delousing. 

•  Landing: 

•  Interoperate  with  airbase  landing/recovery  patterns  and  procedures. 

5.6.1.3  Applicability  of  AI  Techniques  to  Mission  Requirements 

Various  AI  methods  were  examined  in  order  to  determine  their  relative  merits  and  particular  ‘skills’, 
when  applied  to  the  types  of  activity  necessary  for  the  autonomous  mission  tasks  identified  above 
(see  Table  5-2). 


5-38 


RTO-TR-HFM-078 


ARTIFICIAL  COGNITION  AND  CO-OPERATIVE  AUTOMATION 


Table  5-2:  Relative  Merits  of  Al  Techniques  when  Applied  to  UAV  Mission  Management  Tasks 


Fuzzy  Systems 

GAs 

KBS 

NN 

CBR 

Hybrid  systems 

MPS1 

Scheduling/planning 

M 

M 

M 

M 

Y 

y 

Y 

Decision  support 

Y 

Y 

M 

y 

Diagnostics 

Y 

Y 

M 

Y 

y 

Risk  analysis 

Y 

Y 

M 

Y 

Y 

y 

Data  analysis 

M 

Y 

M 

Y 

M 

y 

Monitoring 

Y 

Y 

Y 

M 

y 

Optimisation 

M 

M 

Y 

y 

Interpretation 

Y 

Y 

M 

M 

y 

Classification 

Y 

M 

Y 

Y 

M 

y 

Control  of  systems 

Y 

M 

Y 

Y 

Mission  Planning  Systems. 


Legend 

Y 

Yes  -  highly  applicable 

M 

■ 

Maybe  -  slightly  applicable 

No  -  inapplicable 

The  following  table  (Table  5-3)  compares  the  relative  merits  of  these  AI  techniques  to  the  mission  tasks 
above,  giving  the  tasks  fully  autonomous  UAV  or  UCAV  would  have  to  undertake,  and  which  AI  method, 
or  combination  of  methods  would  best  be  suited.  These  are  then  linked  to  various  processes  (outlined  in 
Section  4  below,  and  see  [4]  for  more  details)  forming  an  example  UAV/UCAV  architecture  (based  on  the 
Sharp  control  and  decisional  architecture  for  autonomous  vehicles  [5]).  The  result  is  the  identification  of 
which  technique  or  combination  of  techniques  is  best  suited  to  each  process. 


RTO-TR-HFM-078 


5-39 


ARTIFICIAL  COGNITION  AND  CO-OPERATIVE  AUTOMATION 


ORGANIZATION 


Table  5-3:  Recommended  Al  Techniques  for  Identified  UAV  Autonomy  Requirements,  and  Mapping  to  Processes 


Action 

Capability 

Mission  Type 

AI  task  for  Autonomous  UCAV 

AI  Technique 

Process 

Autonomous 

Navigation 

Routing  to  pre-defined  target 

All 

Route/Mission  Planning,  Risk  Analysis,  Optimisation 

MPS 

Route  Planner 

Route  re-planning 

Mission  Planning 

MPS 

Route  Planner 

Mission  re-planning 

Mission  Planning 

Hybrid  (KBS  &  MPS) 

Mission  Manager 

Surveillance 

RASP  from  own  sensor  /  datalink  ->  SA 

All 

Monitoring,  Interpretation,  Classification,  Data  Analysis, 
Optimisation 

KBS 

Situational  Awareness, 
Mission  Manager 

Mission/payload  sensor  management 

Scheduling,  Control,  Data  Analysis 

Fuzzy 

Mission  Manager, 
Situational  Awareness 

Threat 

avoidance 

Threat  Detection 

All 

Monitoring,  Interpretation,  Classification, 

Risk  Analysis 

KBS 

Mission  Manager 

Threat  Identification 

KBS 

Mission  Manager 

Threat  avoidance  through  Route/Mission  re¬ 
planning 

Control,  Route/Mission  Planning 

Hybrid  (KBS  /  Fuzzy  & 
MPS) 

Mission  Manager, 

Route  Planner 

Destructive 

Target  Detection 

SEAD,  Strike, 
TCT 

Monitoring,  Interpretation,  Classification,  Risk  Analysis 

KBS 

Mission  Manager 

Real-time  Target  Identification 

Classification,  Data  Analysis 

Hybrid  (KBS  /  Fuzzy  & 
NN) 

Mission  Manager 

Target  Acquisition 

Classification 

Hybrid  (KBS  /  Fuzzy  & 
NN) 

Mission  Manager 

Weapons  release 

Control 

NN 

Mission  Manager, 

Flight  Management 

Battle  Damage  Assessment 

Interpretation,  Classification 

Hybrid  (KBS  /  Fuzzy  & 
NN) 

Mission  Manager 

Interoperability 

Communications  to  command  centre  (maintain 
datalinks,  etc.) 

All,  esp. 

Comms  Relay, 
ISTAR,  TCT 

Data  Analysis,  Monitoring 

Hybrid  (NN  &  Fuzzy) 

Mission  Manager, 
Situational  Awareness 

Integration  into  battlespace  infrastructure 

Data  Analysis 

KBS 

Mission  Manager, 
Situational  Awareness 

Fly  as  part  of  a  package 

SEAD,  Strike 

Optimisation,  Control,  Risk  Analysis 

Hybrid  (KBS  /  Fuzzy  & 
NN) 

Mission  Manager, 
Situational  Awareness, 
Flight  Management 

Interoperate  with  Airbase  /  carrier  patterns  and 
procedures,  ATC 

All,  esp. 

Maritime 

Control,  Planning,  Interpretation,  Data  Analysis 

Hybrid  (KBS  /  Fuzzy  & 
NN) 

Mission  Manager, 

Flight  Management, 
Situational  Awareness 

Interoperate  with  other  manned/unmanned 
platforms 

All 

Control 

Hybrid  (KBS  &  NN) 

Mission  Manager, 

Flight  Management, 
Situational  Awareness 

System 

Fault/damage  tolerant  control,  i.e.,  flight 

All 

Control,  Risk  Analysis,  Data  Analysis,  Diagnostics 

Hybrid  (NN  &  Fuzzy) 

Flight  Management 

Health  Monitoring/diagnostics 

Monitoring,  Classification,  Diagnostics 

CBR 

Health  Monitor 

D/NAW  capability  (aircraft  &  sensors) 

ISTAR 

Interpretation  (sensor  data) 

NN 

Flight  Management, 
Situational  Awareness 

Airmanship,  e.g.,  fuel  states 

All 

Monitoring,  Diagnostics 

KBS,  CBR 

Airmanship 

Safety  &  Emergency  procedures 

Monitoring,  Risk  Analysis,  Control 

KBS 

Airmanship 
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5.6.1.4  Recommended  Applications  of  AI  Methods 

The  analysis  in  Table  5-3  shows  that  no  single  AI  technique  is  appropriate  for  all  areas  of  autonomous 
operation.  The  requirements  for  all  tasks  are  so  diverse  that  one  method  cannot  be  a  panacea.  A  hybrid  system 
would  offer  the  best  solution,  so  that  a  particular  technique  that  excels  in  one  area  can  be  applied  to  one 
subsystem,  providing  input  into  other  subsystems  functioning  through  one  or  more  different  techniques. 

By  comparing  the  techniques  suited  to  each  task  (which  were  outlined  in  the  section  above),  and  the  process 
that  contains  each  task,  it  becomes  apparent  (in  Table  5-2)  which  technique,  or  combination  of  techniques  is 
most  suited  to  each  process  (see  Figure  5-30  for  a  diagrammatic  representation). 


Fused  sensor  data 


Figure  5-30:  Recommended  Applications  of  AI  Method. 

The  Flight  Management  process  would  be  best  as  a  hybrid  system.  A  neural  network  could  control  the  actual 
platform’s  flight  characteristics,  managed  by  a  fuzzy  controller  receiving  instructions  from  the  Mission 
Manager.  This  might  include  neural  network  for  machine  vision  purposes  when  the  aircraft  is  operating  on  the 
ground  (e.g.,  taxiing  around  obstacles). 

A  Case-Based  Reasoning  system  should  manage  the  Health  Monitoring  process.  The  present  state  of  the 
platform  would  be  compared  to  cases  representing  nominal  operation,  and  differences  used  to  diagnose 
failures,  and  possible  remedies,  in  real  time,  from  a  case  library. 
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The  Airmanship  process  would  be  a  Knowledge  Based  System.  This  would  continually  monitor  the  state  of 
fuel  levels,  weapons  and  sensors,  and  the  position  of  the  aircraft  with  respect  to  flight  levels,  safe  altitudes, 
airspace  restrictions  and  so  on,  advising  the  mission  manager  if  plans  are  feasible  given  current  fuel  levels, 
or  require  re -planning. 

Situational  Awareness  would  be  provided  by  another  knowledge-based  system.  This  would  build  a  RASP 
from  correlation  and  fused  sensor,  datalink,  and  other  information,  abstracting  tracks,  and  advising  the 
mission  manager  on  potential  threats  and  targets. 

A  mission  planning  system  would  be  best  for  the  Route  Planning  process.  The  route  planner  would  take  the 
current  mission  objective  and  abstracted  RASP  from  the  mission  manager  and;  with  knowledge  of  airspace 
regulations,  the  platform’s  capabilities  (from  airmanship  and  health  monitoring  processes),  knowledge  of  the 
terrain  for  masking  purposes,  and  ROE;  produce  a  flight  plan. 

The  Mission  Manager  is  in  overall  control  of  the  aircraft.  This  would  be  use  a  hybrid  technique  with  inputs 
from  all  the  other  processes  feeding  a  KBS  or  fuzzy  system.  A  fuzzy  system  could  cope  better  with  uncertain 
information,  however  this  might  be  undesirable  if  information  is  too  vague.  The  mission  manager  abstracts 
information  from  the  SA  process  into  targets  and  threats,  based  on  mission  objectives,  ROE  and  other 
pertinent  information,  for  the  route  planner,  and  passes  resulting  flight  plans  to  the  control  process 
Any  changes  to  the  environment,  ROE,  mission  objectives,  or  RASP  may  result  in  re -planning  the  mission, 
as  could  advice  from  the  airmanship  or  health  monitoring  processes. 

It  is  important  to  remember  also  that  a  human  will  still  be  in  the  loop  somewhere,  most  likely  monitoring  the 
UAV/UCAV  as  part  of  a  system  of  systems  from  a  ground  station,  and  possibly  taking  tactical  decisions  about 
the  overall  mission  plan.  A  soldier  on  the  ground  may  request  surveillance  imagery,  or  even  Close  Air 
Support.  The  overall  requirement  is  for  flexibility  and  network-enabledness  as  part  of  the  future  integrated 
battlespace. 


5.6.1.5  Conclusions  on  AI  Methods 

This  section  has  examined  the  spectrum  of  available  UAV  platforms  and  underpinning  technologies  in  the 
context  of  potential  scenarios  in  order  to  discuss  the  relative  merits  of  different  Ai  techniques  to  improve 
mission  effectiveness. 

Of  necessity,  at  this  period,  most  effort  has  been  expended  in  platform  configuration  and  power  plants  with 
bandwidth  limitations  initially  appearing  as  the  first  barrier  to  more  widespread  application,  after  the  need  for 
greater  systems  integration. 

A  trade  off  then  emerges  of  bandwidth  requirements  for  transmission  against  onboard  processing  for  greater 
autonomy,  and  so  reducing  reliance  on  transmitted  data. 

Autonomy  is  often  seen  as  a  panacea  to  address  the  bandwidth  problem  with  AI  techniques  in  the  vanguard 
for  achieving  this  state,  although  reference  to  the  maturity  of  different  AI  techniques  quickly  refocuses 
attention  on  candidate  approaches  for  down  selection  for  any  practical  demonstrator. 

Autonomy  and  automation  are  not  synonymous,  and  examples  are  offered  in  the  report  that  illustrate  the 
salient  differences  and  the  associated  impact  on  elements  of  any  system. 
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Tabulation  of  the  strengths  and  weaknesses  of  different  AI  techniques  indicate  the  potential  application  area  to 
best  exploit  the  relevant  merits  and  maturity  levels  of  each  technique. 

Contrary  to  widespread  beliefs  held  by  non-specialists,  this  study  indicated  only  limited  potential  for  neural 
networks,  genetic  algorithms  and  fuzzy  systems  in  UAVs,  except  as  a  component  of  more  comprehensive 
hybrid  solutions.  Fuzzy  systems  and  neural  networks  were  seen  as  predominantly  of  use  in  control  and 
classification  applications. 

Knowledge-based  systems  were  seen  as  essential  for  monitoring,  interpretation  and  data  analysis,  that  is, 
the  less  autonomous  aspects  of  missions  such  as  diagnostics  and  decision  support. 

The  application  of  specific  AI  techniques  was  found  to  be  largely  dependent  on  the  level  of  autonomy 
envisaged.  Higher  levels  of  autonomy  suggested  more  hybrid  systems  to  exploit  the  complementary 
capabilities  of  different  AI  technologies. 

From  the  systematic  consideration  of  the  relative  merits  of  different  AI  approaches  for  each  UCAV  mission 
requirement,  it  became  possible  to  identify  which  AI  technique,  or  combination  of  techniques  are  best  suited 
to  each  element  of  the  system. 

Conversely,  if  different  or  varying  levels  of  autonomy  are  considered,  appropriate  AI  techniques  can  be 
identified  and  recommended  to  satisfy  separate  elements  of  the  UCAV/UAV  system. 


5.6.1.6  Recommendations  on  AI  Methods 

This  study  has  outlined  the  relevance  and  maturity  of  AI  techniques  and  technologies  to  address  the  issue  of 
autonomy  in  future  UAVs  and  UCAVs.  As  a  result  of  the  process  of  breaking  down  missions  and  roles  to 
address  the  required  functionality  it  became  evident  that  no  single  technique  was  a  panacea  for  complete 
platform  autonomy,  however  certain  methods  can  be  applied  to  particular  areas,  which  might  be  self- 
contained  should  the  goal  of  complete  autonomy  be  judged  unobtainable  or  undesirable. 

The  most  attainable  solution  would  involve  a  complex  hybrid  architecture,  utilising  several  methods  to 
address  particular  areas,  in  self-contained  elements  or  as  a  coherent  whole.  If  a  fully  or  even  partially 
autonomous  air  vehicle  is  the  goal,  then  much  further  development  will  be  required  to  refine  potential 
architectures  in  light  of  implementation  issues. 

On-  and  off-board  diagnostics  and  health  monitoring  could  be  achieved  by  a  case-based  reasoner. 
A  knowledge-based  system  could  address  airmanship  tasks.  Autonomous  flight  management  might  be 
accomplished  with  a  hybrid  system  comprised  of  a  controlling  fuzzy  system  fed  by  a  neural  network. 
A  knowledge-based  system  is  applicable  to  managing  situational  awareness,  with  dynamic  route  planning  an 
re-planning  undertaken  by  a  mission  planning  system. 

Finally,  a  hybrid  system  could  form  the  mission  management  executive,  based  on  a  KBS  or  fuzzy  expert 
system,  fed  by  inputs  from  the  other  elements. 

There  is  a  vital  need  to  consider  more  closely  the  actual  architecture  of  a  proposed  fully-autonomous  UAV  in 
order  to  gain  a  better  understanding  of  the  particular  systems  that  are  likely  to  be  implemented.  Then,  and  only 
then  can  a  deeper  look  can  be  taken  at  which  AI  techniques  are  most  applicable  to  each  subsystem;  this  report 
identifies  the  areas  that  best  suit  the  relative  ‘skills’  of  certain  AI  methods. 
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Any  realistic  recommendation  would  be  time  and  funding  dependent.  Shorter  timescales  and  minimal  funding 
suggest  applications  limited  to  diagnostic  advisors,  whilst  longer  timescales  and  more  generous  funding  make 
possible  hybrid  AI  solutions  for  applications  requiring  full  autonomy. 

AI  techniques  could  also  be  more  confidently  employed  in  areas  where  less  capability  is  required.  This  report 
has  identified  some  of  the  many  issues  influencing  the  path  to  full  autonomy,  demonstrating  the  magnitude 
and  complexity  of  the  obstacles  to  be  overcome.  An  interim  solution  could  employ  intelligent  tools  to  aid  the 
operators  of  UAVs  and  UCAVs,  both  ground  and  air  controlled,  to  enable  them  to  utilise  many  more  vehicles 
than  at  present,  whilst  working  in  a  supervisory  battle  management  role,  as  opposed  to  a  pilot  role. 
AI  technologies  employed  in  operator  decision  support,  and  on-  and  off-board  diagnostics  systems  could  go  a 
long  way  towards  meeting  these  requirements. 
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5.6.2  Intelligent,  Adaptive  Help  System  Design 

This  contribution  describes  a  comprehensive  approach  to  developing  decision  support  systems  for  providing 
intelligent,  adaptive  aiding  to  users.  The  approach  is  guided  by  the  use  of  a  knowledge  design  and  engineering 
methodology  that  combines  elements  of  the  CommonKADS  and  IDEF  methods,  Explicit  Models  Design  and 
Perceptual  Control  Theory.  The  following  sections  describe  how  those  individual  components  should  be  used 
in  constructing  an  intelligent  help  system. 
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5.6.2.1  The  CommonKADS  Methodology 

“Intelligent”  software,  such  as  that  required  for  a  truly  adaptive  help  system,  is  built  using  knowledge-based 
systems  (KBS),  which  imitate  human  reasoning.  Such  systems  are  typically  composed  of  formal 
representations  of  knowledge  about  a  particular  domain  and  a  mechanism  that  enables  the  system  to  “reason” 
about  that  knowledge  through  the  application  of  inferences.  This  section  examines  the  CommonKADS 
management  and  engineering  methodology  for  analysing  and  designing  knowledge-based  systems  and  the  role 
of  that  approach  in  developing  help  systems. 

The  CommonKADS  methodology  provides  a  step-by-step  approach  to  analysing  a  given  problem  domain. 
Questions  are  posed  at  each  stage  through  standard  “worksheets,”  which  provide  a  systematic  tool  for 
documenting  both  questions  and  answers  concerning  key  issues.  Individual,  numbered  guidelines  are  also 
offered  to  assist  with  the  construction  of  more  detailed  components  of  the  models. 


5.6.2.2  Application  of  CommonKADS  to  Help  System  Design 

The  process  of  applying  CommonKADS  to  build  a  help  system  involves  the  specification  of  the  following  six 
models: 


•  Organisation  Model; 

•  Task  Model; 

•  Agent  Model; 

•  Knowledge  Model; 

•  Communication  Model;  and 

•  Design  Model. 

5. 6. 2. 2. 1  Organisation  Model 

The  Organisation  Model  is  assembled  during  the  initial  feasibility  and  assessment  phase  of  a  CommonKADS 
analysis.  The  primary  emphasis  of  that  phase  is  to  examine  organisational  or  business  processes  that  could 
benefit  from  the  implementation  of  a  knowledge  system.  Coupled  with  that  is  the  subsequent  feasibility 
analysis  that  weighs  the  costs  associated  with  such  an  implementation  against  the  projected  benefits. 
Thus,  a  system  must  be  sufficiently  knowledge-intensive  to  warrant  its  implementation  using  CommonKADS. 
Worksheet  questions  help  to  identify  the  structure  of  the  organisation,  relevant  processes,  people  and 
resources  involved  and  knowledge  assets  required. 

It  should  be  noted  that  the  authors  of  the  CommonKADS  approach  emphasise  the  importance  of  an  initial 
phase  in  which  a  systematic  analysis  of  the  organisation  is  conducted.  This  points  to  situations  where  applying 
the  methodology  has  revealed  needs  for  a  knowledge  system  that  are  substantially  different  from  those 
projected  at  the  outset  of  analysis. 

5. 6. 2. 2. 2  Task  Model 

The  Task  Model  in  CommonKADS  is  created  primarily  as  part  of  the  organisational  analysis  and  feasibility 
assessment  and  focuses  on  high-level  tasks  and  goals  of  agents  in  the  system.  Tasks  should  be  examined  with 
a  view  to  identifying  those  that  are  sufficiently  knowledge-intensive  and  that  would  benefit  from  the 
implementation  of  a  knowledge-based  system. 
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To  create  an  explicit  representation,  tasks  and  sub-tasks  should  be  represented  using  flow  diagrams  from  the 
Unified  Modelling  Language  (UML).  UML  has  been  adopted  by  the  developers  of  CommonKADS  as  the 
standard  technique  for  schematically  depicting  activity,  state  and  class  diagrams  that  can  represent  a  wide 
range  of  concepts  such  as  data  flow,  inferences  and  task  structures. 

Construction  of  the  Task  Model  involves  the  creation  of  a  task  hierarchy,  in  which  tasks  are  decomposed  into 
sub-tasks.  Questions  posed  during  Task  Model  creation  relate  to  constituent  sub-tasks,  the  agents  and  objects 
involved,  the  timing  of  the  task  and  knowledge  required  for  performing  it.  That  information  can  then  be 
referred  to  later  during  knowledge  model  construction. 

Although  CommonKADS  specifies  that  UML  be  used  for  representing  task  hierarchies,  there  are  limitations 
to  UML’s  expressiveness  such  that  a  complement  to  UML  be  considered  in  implementing  help  systems  for 
certain  applications.  The  limitations  relate  to  the  representation  of  temporal  constraints,  such  as  concurrency 
of  tasks  and  their  performance  in  real-time.  In  designing  help  systems  for  applications  in  which  precise 
timings  of  events  is  critical,  it  is  recommended  that  the  IDEF3  language  be  used  for  the  graphical 
representation  of  tasks  and  sub-tasks  (see  the  “IDEF3”  section,  below). 

A  formal  statement  of  tasks  and  sub-tasks  will  help  to  identify  responsibilities  that  can  be  carried  out  by 
software  agents.  Such  tasks  include: 

•  Monitoring  the  user’s  activities; 

•  Inferring  the  user’s  immediate  goals  and  higher-order  intentions;  and 

•  Generating  system  plans  to  assist  the  user  in  the  most  effective  way  given  current  circumstances. 

Identification  of  those  tasks  should  be  performed  in  consultation  with  subject-matter  experts. 

It  should  be  noted  that  the  creation  of  a  task  hierarchy  is  fundamental  to  several  of  the  approaches  that 
comprise  the  integrated  methodology,  including  Explicit  Models  Design  and  Perceptual  Control  Theory. 
Task  hierarchies  are  discussed  further  in  sections  that  follow  (see  the  “Explicit  Models  Design”  and 
“Perceptual  Control  Theory”  sections,  below). 

5. 6.2.23  Agent  Model 

Like  the  two  models  already  described,  the  Agent  Model  is  developed  during  the  initial  feasibility  phase. 
It  is  used  to  identify  the  participants  in  the  itemised  tasks  so  that  their  responsibilities  can  be  incorporated  into 
any  resulting  knowledge  system.  That  process  also  assists  with  identifying  expert  sources  of  knowledge  that 
can  be  useful  in  supplying  information  and  providing  rules  for  the  knowledge  base. 

Construction  of  the  Agent  Model  involves  examining  the  tasks  in  which  each  agent  is  involved,  the  other 
agents  with  which  it  communicates  and  knowledge  that  is  required  to  complete  its  tasks. 

5. 6. 2. 2. 4  Knowledge  Model 

The  Knowledge  Model  contains  a  detailed  enumeration  of  all  knowledge  required  by  the  system  to  perform  its 
tasks.  Thus,  most  of  the  architecture  of  the  knowledge  system  is  designed  during  the  formulation  of  that 
model. 

The  Knowledge  Model  is  subdivided  into  three  categories:  “domain,”  “inference”  and  “task”  knowledge. 
Domain  knowledge  contains  all  of  the  data  used  by  the  application,  which,  in  object-oriented  terminology, 
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would  correspond  to  the  class  definitions  and  object  instances.  In  the  case  of  a  help  system,  the  domain 
knowledge  should  include  knowledge  relating  to  the  capabilities  and  functions  of  the  software,  including  user 
and  system  tasks  described  in  the  Task  Model.  Domain  knowledge  should  also  include  information  about  the 
external  environment,  as  contained  in  the  EMD  World  Model  (see  the  “Explicit  Models  Design”  section 
below). 

The  second  component  of  the  Knowledge  Model  is  the  inference  knowledge,  which  is  a  collection  of  methods 
that  act  on  the  domain  knowledge.  CommonKADS  provides  a  catalogue  of  inference  templates,  an  approach 
that  has  multiple  advantages.  Creating  methods  that  can  be  applied  generally  allows  them  to  be  reused  readily 
in  other  applications.  Because  they  have  been  applied  in  other  situations,  they  come  pre-tested  and  therefore 
contribute  to  the  overall  reliability  of  the  knowledge  system.  Examples  of  inference  knowledge  in  a  help 
system  would  include  methods  for  inferring  goals  from  actions  or  hypothesising  a  plan  based  on  a  user’s 
actions. 

The  final  component  of  the  Knowledge  Model  is  task  knowledge,  comprising  a  set  of  higher-level  methods 
that  implement  a  hierarchy  of  tasks  and  sub-tasks.  At  the  lowest  level,  the  sub-tasks  make  use  of  methods  in 
the  underlying  inference  knowledge  layer,  which  in  turn  operate  on  the  domain  knowledge.  Task  knowledge 
in  a  help  system  would  include  a  representation  of  the  full  task  hierarchy  derived  in  the  specification  of  the 
EMD  Task  Model  (see  the  “Explicit  Models  Design”  section  below). 

As  with  the  inference  knowledge,  there  are  templates  available  for  task  knowledge  and  they  share  the  same 
benefits  of  reusability  and  reliability.  Templates  that  are  provided  include  those  for  planning,  scheduling  and 
monitoring,  all  of  which  are  useful  in  a  help  system.  Planning  and  scheduling  are  tasks  required  for  plan 
generation  and  monitoring  is  essential  to  plan  recognition  (see  the  section  on  Explicit  Models  Design  below). 

Knowledge  modelling  in  any  context  typically  involves  the  creation  of  an  ontology,  and  CommonKADS  is  no 
exception.  Ontologies  are  formally  specified  frameworks  within  which  knowledge  can  be  represented.  A  chief 
goal  in  producing  an  ontology  is  to  identify  patterns  in  the  knowledge  and  exploit  those  to  produce  a  highly 
organised  and  concise  specification.  One  of  the  main  motivations  for  generating  an  ontology  for  a  particular 
domain  is  to  allow  its  reuse  in  other  applications. 

In  order  to  describe  fully  the  Knowledge  Model  for  a  help  system,  it  will  be  necessary  first  to  design  the 
content  of  the  various  Explicit  Models.  The  section  on  “Explicit  Models  Design”  describes  how  those  models 
are  constructed. 

5. 6. 2. 2. 5  Communication  Model 

The  purpose  of  the  communication  model  is  to  describe  communication  that  must  occur  among  agents  in  the 
knowledge  system.  That  can  include  dialogue  that  is  both  between  the  user  and  system  agents,  and  between 
individual  software  agents. 

Communication  is  broken  down  using  a  transaction  model.  For  each  pair  of  agents  that  must  interact, 
a  communication  plan  should  be  constructed  (usually  represented  using  UML)  that  outlines  the  flow  of 
information  and  decisions  affecting  that  flow.  That  is  decomposed  further  into  a  detailed  itemisation  of 
individual  transactions,  where  each  one  represents  a  message  sent  from  one  agent  to  another.  Each  transaction 
should  be  described  in  terms  of  the  agents  involved,  the  content  of  the  messages  and  knowledge  objects 
exchanged. 
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Standard  patterns  of  communication  are  described  in  CommonKADS,  such  as  the  straightforward  “Ask” 
and  the  associated  “Reply,”  or  slightly  more  complex  exchanges  such  as,  “Require,”  which  can  have  “Agree” 
or  “Reject”  as  responses.  A  library  is  offered  for  those  and  other  standard  modes  of  communication. 

One  type  of  user-system  interaction  in  a  help  system  involves  the  presentation  of  information  by  the  system 
and  possible  acknowledgement  from  the  user.  (In  situations  where  it  would  be  disruptive  to  the  user  to  provide 
explicit  feedback,  the  system  should  infer  through  indirect  means  that  the  user  has  received  the  information.) 
In  addition  to  that  system-initiated  communication,  users  should  be  able  to  request  help  information  from  the 
system,  which  necessitates  a  second  form  of  dialogue.  Interaction  may  also  consist  of  a  clarification  dialogue 
between  user  and  system  agents  whereby  the  system  seeks  information  when  the  user’s  current  intentions  are 
ambiguous,  but  care  must  be  taken  to  avoid  unnecessary  requests  for  communication  with  the  user.  Users  may 
also  seek  clarification  on  system  goals  or  activities. 

The  Communication  Model  also  describes  dialogue  that  occurs  among  system  agents,  which  is  important  in 
the  help  system  to  maintain  co-ordination  among  semi-autonomous  entities.  Agent  interaction  in  the 
Communication  Model  is  revisited  in  the  “Software  Agent  Paradigm”  section  below. 

5. 6. 2. 2. 6  Design  Model 

The  Design  Model  examines  hardware  and  software  issues  related  to  the  construction  of  the  knowledge 
system.  The  aim  is  to  take  the  implementation-independent  specifications  from  the  Knowledge  and 
Communication  Models  and  develop  a  detailed  design  for  constructing  the  software  application,  and  in  the 
process  preserve  the  structure  of  those  models. 

CommonKADS  help  systems  should  be  designed  using  the  Model- View-Controller  (MVC)  architecture. 
In  that  approach,  the  Application  Model  contains  the  rules,  inference  functions,  and  knowledge  bases  that  are 
responsible  for  the  main  functionality  of  the  application.  The  Views  subsystem  provides  external  views  of  the 
data  in  the  application  model,  which  can  be  in  the  form  of  a  user-interface  or  can  also  involve  the  presentation 
of  information  to  an  external  software  system.  The  Controller  handles  the  processing  of  events,  the  triggering 
of  tasks  and  inferences,  and  the  responsibilities  of  the  Communication  Model. 

The  next  design  step  is  identifying  the  target  software  and  hardware  platforms.  It  is  recommended  that 
CommonKADS  systems  be  implemented  in  an  object-oriented  (O-O)  environment. 

Some  suggested  languages  for  implementing  CommonKADS  systems  are  Prolog  and  Java,  but  that  is  not  to 
the  exclusion  of  other  possible  environments  (For  examples  and  source  code,  see  [1]  and  the  CommonKADS 
web  site  at  www.commonkads.uva.nl). 

Once  an  implementation  environment  has  been  selected,  the  final  step  in  constructing  the  Design  Model  is  to 
create  a  detailed  plan  for  implementation  of  the  Application  Model,  Views  and  Controller,  as  well  as  the  tasks, 
inferences  and  domain  knowledge  within  the  Knowledge  Model.  Many  details  of  the  plan  are  dependent  on 
the  chosen  environment. 

CommonKADS  also  includes  guidelines  on  project  management  that  are  designed  to  accommodate  the  unique 
needs  associated  with  knowledge  projects. 
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5.6.2.3  IDEF  Standards 

The  IDEF  (ICAM  Definition)  standards,  like  the  CommonKADS  methodology,  provide  a  set  of  guidelines  for 
analysing  processes,  activities  and  information  needs  within  organisations.  The  IDEF  documents  are  a 
collection  of  numbered  standards,  each  of  which  provides  formal  guidelines  for  analysis  and  design  in  a 
particular  area.  The  two  standards  relevant  to  help  system  design  are  “IDEF3:  Process  Modelling”  and 
“IDEF 5 :  Ontology  Modelling.” 

5. 6.2.3. 1  IDEF 3:  Process  Modelling 

The  CommonKADS  approach  uses  UML  for  the  schematic  representation  of  processes  and  associated  data, 
agents,  tasks  and  inferences.  One  of  the  drawbacks  of  UML  is  that  it  is  inflexible  in  representing  temporal 
relationships  and  constraints  among  those  elements.  Two  important  temporal  concepts  are  synchronisation  and 
real-time  systems.  In  the  case  of  the  former,  a  distinction  is  made  between  synchronous  and  asynchronous 
activities,  i.e.,  those  that  occur  at  the  same  time  contrasted  with  those  that  do  not,  respectively. 
That  distinction  can  have  important  consequences  for  the  system  being  modelled,  where,  for  example,  tasks 
that  can  be  carried  out  synchronously  can  shorten  the  total  time  required  to  complete  a  procedure  and  thereby 
increase  the  efficiency  of  the  system.  When  modelling  real-time  systems,  much  more  attention  must  be  paid  to 
the  precise  times  at  which  events  occur,  not  simply  their  relative  occurrence,  and  that  can  have  direct  bearing 
on  whether  or  not  the  modelled  system  will  perform  as  intended. 

Unlike  UML,  IDEF3  permits  flexible  modelling  of  temporal  concepts.  For  example,  symbolic  representations 
exist  for  depicting  whether  multiple  activities  are  synchronous  or  asynchronous  and  whether  all  activities  must 
be  complete  before  the  next  steps  in  the  process  can  continue. 

Modelling  of  real-time  systems  is  possible  using  IDEF3’s  elaboration  language,  which  allows  symbolic 
representations  of  complex  constraints  that  cannot  be  depicted  using  the  schematic  language  alone. 
The  language,  based  on  a  subset  of  the  Knowledge  Interchange  Format,  permits  formal  logical  representations 
of  process  constraints  and  allows  precise  specification  of  event  timings  and  durations. 

Because  IDEF3  has  full  flexibility  for  temporal  modelling,  and  because  it  has  been  in  use  for  a  long  time, 
IDEF3  is  recommended  for  use  in  help  system  development  for  applications  with  precise  timing  needs. 
In  future,  it  may  be  possible  to  achieve  greater  flexibility  for  representing  temporal  constraints  using  UML. 
The  upcoming  release  of  the  language  (UML  2.0)  promises  to  offer  more  of  such  capabilities,  and  the  addition 
of  the  Object  Constraint  Language  (OCL)  to  UML  also  features  added  support  for  temporal  modelling. 

5. 6. 2. 3. 2  IDEF 5:  Ontology  Modelling. 

Another  IDEF  standard  that  is  relevant  to  help  system  design  is  IDEF5,  which,  like  CommonKADS,  provides 
specifications  for  ontology  modelling.  CommonKADS  provides  techniques  for  developing  ontologies, 
including  guidance  on  expert  knowledge  elicitation,  formal  descriptions  of  concepts,  attributes  and  relations, 
and  a  formal  language  for  representing  ontologies.  Although  those  are  powerful  components,  ontology 
construction  is  explored  in  greater  depth  in  IDEF5,  with  more  guidance  offered  and  more  examples  provided. 
The  IDEF5  approach  has  five  steps: 

•  Organising  and  scoping; 

•  Data  collection; 

•  Data  analysis; 
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•  Initial  ontology  development;  and 

•  Ontology  refinement  and  validation. 

The  organising  and  scoping  phase  examines  the  context  and  purpose  of  the  project,  and  can  make  use  of  the 
material  assembled  during  the  specification  of  the  Organisation,  Task  and  Agent  Models  of  CommonKADS. 
Data  collection  involves  acquiring  raw  data  by,  for  example,  examining  existing  systems  or  eliciting 
knowledge  from  experts.  Data  analysis  attempts  to  refine  the  raw  data  into  a  form  more  usable  in  an  ontology. 
Initial  ontology  development  creates  a  draft  of  the  ontology  which  is  further  developed  in  the  refinement  and 
validation  phase.  The  methodology  divides  each  of  those  five  stages  into  sub-steps  and  provides  detailed 
guidelines  on  how  to  accomplish  each. 


5.6.2.4  Explicit  Models  Design 

Explicit  Models  Design  (EMD)  is  a  development  approach  that  seeks  to  make  explicit  the  knowledge  required 
by  intelligent  software  systems.  The  approach  compartmentalises  software  knowledge  into  five  distinct, 
interacting  models: 

•  Task  Model,  containing  knowledge  (beliefs)  about  tasks  being  performed; 

•  System  Model,  consisting  of  the  system’s  knowledge  (beliefs)  about  itself  and  its  abilities; 

•  User  Model,  comprised  of  knowledge  (beliefs)  relating  to  the  user’s  abilities,  needs  and  preferences; 

•  World  Model,  representing  knowledge  (beliefs)  about  the  world  relevant  to  the  purpose  of  the 
software;  and 

•  Dialogue  Model,  containing  knowledge  (beliefs)  related  to  communication  among  human  and 
software  agents. 

Plan  recognition  and  plan  generation  are  two  additional  processes  that  operate  within  the  EMD  framework  to 
enhance  the  software’s  ability  to  support  the  user.  Plan  recognition  seeks  to  establish  the  current  goals  of  the 
user  in  the  context  of  a  larger  plan.  This  process  also  seeks  to  recognise  goals,  plans  and  actions  in  terms  of 
the  help  that  might  be  required  by  a  user  to  perform  a  task  in  a  more  effective  way.  Plan  generation  is  used  by 
the  system  to  develop  strategies  to  accomplish  its  goals,  which  principally  involve  providing  help  to  the  user. 
Those  techniques,  and  the  individual  Explicit  Models,  are  described  below  in  the  context  of  the  help  system. 

5. 6.2.4. 1  EMD ’s  Contributions  to  the  Help  System 

Within  the  help  system,  EMD  offers  a  means  of  subdividing  the  content  of  the  CommonKADS  Knowledge 
Model  into  components,  described  in  the  following  sections.  Specification  of  all  models  must  be  done  in 
consultation  with  subject-matter  experts. 

5. 6. 2. 4. 2  Task  Model 

The  Task  Model  contains  knowledge  relating  to  the  tasks  being  performed  by  the  user,  represented  as  a 
hierarchy  of  actions,  goals  and  plans.  At  the  lowest  levels  of  the  hierarchy  are  primitive  interface  actions, 
such  as  button  clicks  and  menu  selections.  EMD  recognises  that  each  deliberate  interface  action  carried  out  by 
a  user  is  in  support  of  a  particular  goal  and  that  actions  may  be  expressed  in  the  terminology  of  such  goals. 
For  example,  if  a  user  clicks  an  “OK”  button,  the  system  can  infer  that  the  user’s  goal  was,  “to  click  the  ‘OK’ 
button.”  While  the  system  can  easily  infer  that  low-level  goal  from  the  simple  act  of  clicking  an  OK  button, 
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it  is  typically  much  more  difficult  to  establish  a  higher-level  purpose  unless  additional  actions  are  observed. 
That  process,  which  involves  both  higher-level  goals  and  context  (the  particular  “OK”  button  that  was 
clicked),  is  described  in  the  “Plan  Recognition”  section  below. 

Above  the  primitive  actions  and  their  associated  low-level  goals  in  the  hierarchy  are  higher-level  goals, 
which  can  be  achieved  only  by  satisfying  one  or  more  primitive  goals.  Sufficiently  high-level  goals  are  often 
associated  with  what  are  commonly  known  as  tasks. 

A  path  from  a  terminal  node  of  the  tree  up  to  a  higher-level  goal  constitutes  a  plan  for  accomplishing  that 
goal,  and  there  can  be  many  possible  plans  for  satisfying  a  given  high-level  goal.  Plan  recognition  enables  the 
system  to  determine  which  of  those  plans  a  user  is  pursuing  (see  “Plan  Recognition”  below)  and  plan 
generation  (see  the  “Plan  Generation”  section)  permits  the  system  to  select  a  course  of  action  from  its 
available  plans,  or  to  recommend  a  series  of  actions  for  the  user  to  satisfy  an  inferred  high-level  goal. 

The  tracking  of  user  interface  actions  and  inference  of  associated  goals  provides  the  system  with  a  basis  for 
understanding  what  a  user  is  trying  to  accomplish  and  for  helping  that  user  in  ways  that  are  both  relevant  and 
useful.  The  system’s  ability  to  deduce  user  goals  would  be  an  essential  part  of  any  intelligent  help  system  and 
EMD  provides  effective  methods  for  designing  such  a  system. 

5. 6. 2. 4. 3  System  Model 

The  System  Model  is  composed  of  the  system’s  knowledge  about  itself,  its  abilities  and  the  means  by  which  it 
can  assist  users.  Like  the  Task  Model,  the  System  Model  also  contains  a  goal  hierarchy,  describing  the  tasks, 
goals  and  plans  that  the  system  can  carry  out  in  support  of  the  user.  Those  goals  are  characterised  as  system 
support  goals. 

In  the  help  system,  the  System  Model  task  hierarchy  includes  high-level  goals,  such  as,  “to  assist  the  user,” 
which  would  be  decomposed  into  sub-goals,  such  as  those  associated  with  assuming  control  of  functions  it 
had  been  assigned,  monitoring  system  status  and  helping  the  user  to  complete  his  or  her  tasks. 

5.6.2.4A  User  Model 

The  User  Model  is  comprised  of  knowledge  about  the  user’s  abilities,  needs  and  preferences.  That  information 
is  obtained  in  three  ways: 

•  From  information  volunteered  by  the  user; 

•  From  results  of  system  requests  of  the  user;  and 

•  From  system  monitoring  of  user’s  activities. 

It  is  worth  noting  that  the  system  should  be  able  to  identify  a  user  so  that  it  can  maintain  a  unique  profile  for 
each  user.  Unless  that  can  be  done,  the  system  is  reduced  to  providing  information  that  is  often  too  general, 
repetitive  or  useless. 

Information  volunteered  by  a  user  often  occurs  in  the  context  of  specifying  options  and  preferences  to  the 
system.  It  is  important  that  users  be  able  to  specify  preferences  in  order  to  facilitate  efficient  use  of  the 
software,  and  that  is  especially  useful  in  applications  that  offer  a  large  number  of  features  and  settings. 
One  method  of  providing  that  flexibility  is  through  the  establishment  of  agreements  between  the  user  and 
system  using  the  PACT  approach  (see  “The  PACT  Approach  and  Automation  Levels”  section,  below). 
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There  are  several  ways  in  which  systems  can  construct  user  models  by  explicitly  asking  questions  of  a  user. 
For  example,  if  the  system  has  determined  which  task  a  user  is  pursuing,  it  could  enquire  whether  assistance  is 
needed  in  carrying  out  that  task.  The  system  also  might  ask  if  the  user  is  aware  of  more  efficient  plans  for 
accomplishing  the  task.  Finally,  if  the  system  cannot  determine  a  user’s  current  plan,  it  may  seek  clarification 
on  the  user’s  intentions. 

In  a  truly  intelligent  system,  User  Model  knowledge  is  acquired  indirectly  by  monitoring  user  activities  in  a 
Task  Model.  If  the  system  observes  the  user  carrying  out  significant  and  logical  portions  of  a  particular  plan, 
it  assumes  with  a  fairly  high  degree  of  confidence  that  the  user  understands  that  process.  The  system’s 
confidence  increases  as  the  user  is  observed  to  repeat  the  procedure. 

If  the  system  determines  there  is  a  high  probability  that  a  user  lacks  certain  required  knowledge,  it  could 
signal  a  need  to  offer  the  information.  The  system  must  be  able  to  gauge  the  importance  of  communicating  the 
information  in  order  to  establish  a  method  for  doing  so.  For  example,  if  elements  of  the  total  system  are  at  risk 
of  being  lost,  the  user  likely  would  need  to  be  informed  immediately.  In  contrast,  advice  on  carrying  out  tasks 
efficiently  might  not  be  presented  until  the  user  has  either  completed  the  current  task  or  finished  the  session. 

5. 6.2.4. 5  World  Model 

The  World  Model  contains  the  software’s  knowledge  about  the  external  world:  the  objects  that  exist  in  the 
world,  their  properties  and  the  rules  that  govern  them.  Those  rules  can  take  on  a  wide  variety  of  forms,  such  as 
physical  (e.g.,  the  physical  properties  of  objects  in  a  workspace)  and  psychological  (e.g.,  rules  describing 
human  behaviour  in  situations  of  high  cognitive  workload). 

The  process  of  knowledge  elicitation  required  to  construct  a  World  Model  involves  creating  a  formal 
representation  of  information  gleaned  from  subject-matter  experts.  The  resulting  compendium  of  domain- 
specific  knowledge  often  includes  useful  information  that  experts  have  learned  through  experience  is  not 
contained  in  existing  standard  operating  procedures.  That  knowledge  can  then  act  as  feedback  in  the  review  of 
those  procedures. 

Knowledge  stored  in  the  World  Model  will  form  the  basis  of  tutorials  and  “wizards”  guiding  the  user’s  pursuit 
of  goals.  That  guidance  may  include  providing  recommendations  on  creating  and  manipulating  objects, 
as  well  as  accessing  and  entering  data,  activities  that  frequently  require  an  understanding  of  how  elements 
within  the  software  interrelate  with  those  in  the  external  world.  That  knowledge  will  be  structured  to  support 
wizards  that  are  adaptable  to  the  range  of  domains  for  which  help  is  to  be  offered. 

5. 6. 2. 4. 6  Dialogue  Model 

The  Dialogue  Model  contains  knowledge  about  the  manner  in  which  communication  takes  place  among  user 
and  system  agents.  Such  communication  would  involve  interaction  between  the  user  and  system  and  among 
other  system  agents. 

Because  there  are  many  system  agents  in  a  help  system,  it  is  essential  to  specify  a  common  language  and 
protocol  for  them  to  communicate.  In  addition  to  that,  effective  user-system  and  system-system  collaboration 
will  require  the  explicit  representations  of  communication  provided  by  the  Dialogue  Model. 

5.6.2.5  Plan  Recognition 

The  ability  to  recognise  user  plans  is  an  important  element  in  EMD  and  enhances  the  system’s  “awareness”  of 
what  a  user  is  trying  to  accomplish  so  that  it  can  decide  how  best  to  offer  assistance.  It  is  the  infrastructure  of 
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intelligent,  adaptive  aiding.  The  COLLAGEN  plan  recognition  approach  is  recommended  for  help  system 
implementation. 

COLLAGEN  uses  a  “recipe”  approach  whereby  plans  that  the  user  may  be  pursuing  are  assembled  from  plan 
fragments.  When  an  interface  action  is  observed  by  the  system,  the  fragments  are  assembled  to  form  alternate 
sets  of  plans  that  might  explain  why  the  user  performed  that  action.  There  may  be  many  possible  sets. 
As  further  user  actions  are  observed,  the  number  of  possible  sets  that  encompass  that  series  of  actions 
diminishes,  leading  to  a  more  accurate  determination  of  the  user’s  true  plan. 

Plan  recognition  should  occur  in  the  context  of  a  system  of  rules  to  classify  user  activities  according  to  a  set  of 
criteria  that  identify  whether  the  user  is  carrying  out  the  current  task: 

•  Correctly; 

•  Completely; 

•  Consistently; 

•  Efficiently;  and 

•  Safely. 

Violations  of  those  criteria  should  signal  a  possible  opportunity  for  the  system  to  help  the  user  (See  “The  Five- 
Part  Taxonomy  for  Plan  Recognition”  section,  below). 

5. 6. 2. 5.1  Plan  Generation 

Plan  generation  is  the  process  by  which  the  system  develops  strategies  for  accomplishing  its  goals  to  assist  the 
user.  It  is  based  on  System  Model  knowledge  of  a  hierarchy  of  available  support  goals  and  plans,  Task  Model 
knowledge  of  the  user’s  current  goals  and  plans  and  User  Model  knowledge  of  the  operator’s  preferences  and 
abilities. 

Plan  generation  in  the  help  system  would  seek  to  construct  the  most  effective  plans  for  offering  help  to  the 
user,  e.g.,  by  displaying  a  help  message  immediately  or  by  waiting  until  a  suitable  time  to  present  the 
information  with  less  disruption  to  the  user.  System  generation  of  plans  also  will  depend  on  the  selected  level 
of  automation. 

The  processes  of  plan  recognition  and  plan  generation  also  can  be  associated  with  activities  in  Perceptual 
Control  Theory  (PCT),  whereby  plan  recognition  is  associated  with  the  perceptual  input  to  hierarchies  of 
system  control  loops  and  plan  generation  forms  the  behavioural  output  of  similar  hierarchies.  Those  parallels 
bridge  the  EMD  and  PCT  techniques  in  the  help  system  (see  the  “Perceptual  Control  Theory”  section  below). 

5.6.2.6  Feedback 

The  concept  of  feedback  is  important  in  EMD  for  establishing  mutual  understanding  and  support  between  the 
user  and  the  system,  enabling  one  agent  to  inform  another  of  its  goals,  plans  and  knowledge.  Feedback  can 
assume  multiple  forms,  both  explicit  and  subtle. 

Explicit  feedback  can  occur  in  the  form  of  dialogues  among  agents.  For  example,  the  system  may  ask  a  user 
whether  he  or  she  is  familiar  with  a  particular  concept.  The  user’s  response  constitutes  feedback  to  the  system, 
providing  knowledge  for  the  User  Model  and  therefore  enabling  the  system  to  offer  more  appropriate 
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assistance.  Similarly,  a  user  might  ask  the  system  to  explain  its  last  action,  particularly  if  that  action  was 
performed  on  the  system’s  own  initiative.  The  response  from  the  system  is  feedback  that  gives  the  user  a 
better  understanding  of  how  the  software  operates.  The  communication  of  explicit  feedback  among  agents  is 
governed  by  the  Dialogue  Model,  which  must  be  designed  to  support  exchanges  among  agents  involving  the 
provision  of  feedback. 

A  less  overt  form  of  feedback  arises  in  the  form  of  system  support  goals  and  user  goals.  For  example, 
if  a  user’s  goal  is  to  open  a  window  in  the  software  interface,  the  system  will  have  a  corresponding  support 
goal  to  display  that  window.  The  display  of  that  window  constitutes  feedback  to  the  user  that  the  goal  of 
opening  the  window  in  the  virtual  environment  has  been  achieved.  Representations  of  user  goals  also  are 
important  forms  of  feedback  since  they  are  the  primary  means  by  which  the  system  knows  and  learns  about  a 
user.  Detecting  user  goals  is  detecting  feedback  in  that  those  goals  implicitly  inform  the  system  of  a  user’s 
plans,  abilities  and  preferences. 

5.6.2.7  Perceptual  Control  Theory 

Another  theoretical  approach  recommended  for  use  in  help  system  design  is  Perceptual  Control  Theory  (PCT). 
IDEF  and  CommonKADS  methodologies  provide  frameworks  for  approaching  the  design  and  implementation 
of  the  system.  PCT  and  Explicit  Models  Design  (EMD)  have  the  potential  to  influence  how  the  system 
functions  within  that  implementation  framework.  This  includes  how  it  determines  the  goals  a  user  is  trying  to 
achieve,  the  plans  for  achieving  those  goals  and  how  it  can  assist  the  operator  most  effectively.  The  techniques 
complement  one  another  and  together  provide  an  opportunity  to  form  a  comprehensive  approach  that 
combines  the  strengths  of  the  individual  components. 

PCT  is  founded  on  notions  from  control  theory,  in  which  closed-loop,  negative-gain,  feedback  systems  can  be 
used  to  build  powerful  models  of  goal-directed  behaviour  and  to  implement  complex  systems.  The  ways  in 
which  PCT  contributes  to  help  system  design  fall  into  two  basic  categories: 

•  Performing  hierarchical  goal  analyses;  and 

•  Using  PCT  principles  in  the  algorithms  of  the  system. 

5. 6.2. 7. 1  Hierarchical  Goal  Analysis  of  Help  System  Tasks 

A  method  has  been  proposed  for  Hierarchical  Goal  Analysis  (HGA)  using  principles  from  PCT,  and  that 
technique  has  the  potential  to  produce  a  robust  and  complete  task  and  goal  decomposition  for  help  system 
implementation.  That  approach  to  systems  analysis  examines  the  goal  of  an  agent  as  a  desire  to  achieve  a 
certain  perception. 

The  PCT-based  HGA  technique  permits  two  additional  analyses  to  be  performed.  The  first  is  a  stability 
analysis  that  identifies  possible  conflicts  among  multiple  human  and  machine  agents  acting  on  the  same 
system.  The  second  is  an  analysis  of  information  flow  up  the  hierarchy,  which  can  influence  feedback  in  the 
system,  affecting  error-correction  at  higher  levels.  Traditional  HGA  systems  analysis  techniques  consider  only 
the  downward  flow  of  information  in  the  hierarchy.  In  the  case  of  a  help  system,  it  is  important  that 
information  in  the  form  of  system  and  user  goals  and  sub-goals  be  able  to  flow  freely  in  both  directions. 
Stability  and  information  flow  analyses  could  contribute  to  the  generation  of  a  robust  goal  hierarchy  for  the 
help  system. 

A  parallel  can  be  drawn  with  the  flow  of  information  within  the  action,  goal  and  plan  hierarchy  of  Explicit 
Models  Design.  When  a  user  is  performing  actions  in  the  interface,  there  is  a  downward  flow  of  information 
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to  which  the  system  responds  with  system  support  goals  that  feed  back  upward.  From  the  point  of  view  of  the 
system,  its  goals  are  met  with  feedback  from  the  user  flowing  in  the  opposite  direction.  Those  properties  make 
the  PCT-based  HGA  approach  equally  applicable  to  a  parallel  analysis  within  the  EMD  hierarchies. 

5. 6.2. 7.2  Control  Loop  Hierarchies 

PCT  systems  can  be  implemented  as  a  hierarchy  of  control  loops,  wherein  the  output  of  the  higher  levels 
determines  the  reference  signals  at  the  levels  below  and  the  perceptions  at  lower  levels  feed  the  inputs  at  the 
levels  above. 

To  implement  control  loops  in  the  help  system,  the  loops  need  to  be  assigned  a  hierarchy  of  goals,  modelled 
largely  at  that  level  and  described  in  the  “Hierarchical  Goal  Analysis  of  Tasks  in  the  Help  System”  section 
above.  Those  control  loops  have  as  input  the  actions  carried  out  by  the  user.  The  actions  serve  as  a  basis  for 
system  perceptions  about  the  user’s  need  for  assistance  and  that  constitutes  feedback  in  the  loops. 

5.6.2.8  Help  System  Goals  and  Sub-Goals 

If  a  high-level  goal  of  the  system  is,  “to  have  the  perception  (to  believe)  that  the  user  is  performing  the  current 
task  sufficiently  well,  that  is,  in  a  way  that  requires  no  intervention  by  the  system,  a  control  loop  would  need 
to  be  monitoring  activities  at  the  interface  (perceptual  input)  to  detect  the  satisfaction  of  that  goal.  Support  for 
such  a  control  loop  means  that  the  system  must  infer  and  represent  a  belief  that  the  user  is  engaged  in  a 
particular  task  based  on  perceived  user  activity  at  the  interface.  In  order  to  infer  such  a  belief,  the  system  must 
have  knowledge  of  the  structure  of  goals  and  sub-goals  necessary  for  operators  to  perform  the  task. 

The  high-level  system  goal  can  be  further  decomposed  into  sub-goals  concerning  types  of  user  activity  that 
would  suggest  to  the  system  whether  some  form  of  assistance  is  necessary.  System  control  loops  would 
identify  a  need  to  offer  help  when  it  is  perceived  that  the  user  is  not  performing  a  task: 

•  Correctly; 

•  Completely; 

•  Consistently; 

•  Efficiently;  and 

•  Safely. 

That  gives  rise  to  five  sub-goals,  each  with  its  own  hierarchy  of  sub-goals.  The  system’s  decision  to  offer  help 
is  based  in  part  on  an  assessment  of  user  needs  according  to  whether  tasks  are  being  carried  out  in  compliance 
with  the  five  criteria  above.  For  more  detail,  see  the  section  on  “The  Five-Part  Taxonomy.” 

In  an  adaptive  interface,  system-generated  plans  to  assist  users  should  be  incorporated  into  the  behavioural 
components  of  control  loops.  Plan  generation  mechanisms  should  examine  perceptual  error  signals  and 
formulate  appropriate  behavioural  responses  to  correct  them.  The  plan  generator  should  take  into  account  the 
magnitude  of  the  error  signal  in  determining  the  optimal  behaviour  for  providing  assistance  under  the 
circumstances,  whether  it  be  automatic  or  involve  querying  the  operator  on  how  to  proceed. 

5.6.2.9  Software  Agent  Paradigm 

An  autonomous  software  agent  is  a  programme  with  the  ability  to  sense  its  environment  and  to  act  on  that 
environment  over  time  to  achieve  some  purpose  and  to  influence  what  it  will  sense  in  the  future.  Further 
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distinctions  among  agents  can  be  made  based  on  their  behaviour,  for  example,  communicative  agents  can 
interact  with  other  agents  or  people;  adaptive  (or  learning)  agents  can  alter  their  behaviour  based  on  past 
experience;  and,  mobile  agents  can  move  themselves  to  other  machines. 

The  agent-oriented  development  paradigm  offers  several  advantages  that  were  not  addressed  by  earlier  object- 
oriented  approaches,  including: 

•  Increased  modularity; 

•  Enhanced  reusability; 

•  Improved  organisational  effectiveness; 

•  Increased  speed; 

•  Increased  reliability;  and 

•  Better  distribution. 

5. 6. 2. 9. 1  Agents  in  CommonKADS 

The  CommonKADS  (CK)  methodology  is  entirely  consistent  with  the  use  of  software  agents.  The  Agent 
Model  in  the  methodology  allows  for  systems  with  multiple  human  and  software  components. 

(See  “The  CommonKADS  Methodology”  section,  above) 

A  Multi-Agent  System  extension  of  the  CommonKADS  methodology  (MAS-CommonKADS)  has  been 
proposed  and  is  recommended  for  use  in  help  system  development.  The  methodology  was  developed  to  add 
specific  agent-related  constructs,  including  those  associated  with: 

a)  Inter-agent  communication; 

b)  The  division  of  tasks  among  individual  agents;  and 

c)  The  implications  for  implementation  of  multi-agent  systems. 

The  Communication  Model  in  CommonKADS  is  primarily  focussed  on  interaction  between  the  user  and 
individual  system  agents,  with  little  attention  paid  to  communication  among  the  system  agents  themselves. 
To  address  that  issue,  MAS-CommonKADS  incorporates  a  Co-ordination  Model,  which  specifies  how 
messages  are  exchanged,  what  communication  protocols  are  used  and  what  abilities  each  agent  has  for 
interacting  with  others.  Because  of  the  many  commonalities  between  the  Communication  and  Co-ordination 
Models,  the  latter  should  be  treated  as  an  entity  within  the  former. 

The  division  of  labour,  allocated  tasks,  among  agents  is  an  important  consideration  in  the  MAS- 
CommonKADS  approach.  The  physical  locations  of  agents  and  connections  among  them  can  influence  the 
assignment  of  responsibilities  to  each  component.  Task  allocation  also  affects  the  knowledge  requirements  of 
each  agent. 

The  multi-agent  approach  further  influences  the  construction  of  Design  Model  specifications.  Consideration 
must  be  given  to  network  facilities  and  transfer  protocols  according  to  hardware  and  software  constraints. 
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5. 6.2. 9. 2  Agents  in  Explicit  Models  Design 

Explicit  Models  Design  (EMD),  described  above,  also  supports  multi-agent  system  development. 
EMD  recognises  the  roles  of  the  User  and  System  as  agents  and  can  accommodate  both  multiple  human  users 
and  system  agents,  each  represented  by  its  own  User  or  System  (Agent)  Model. 

The  EMD  Dialogue  Model  provides  a  framework  for  describing  communication  among  multiple  human  and 
system  software  agents.  That  model  allows  for  various  modes  of  communication,  including  the  following, 
relevant  to  the  help  system: 

•  A  system  providing  help  information  and  requesting  acknowledgement  from  a  user; 

•  A  system  prompting  a  user  for  clarification  feedback  about  that  user’s  goals;  and 

•  A  multi-agent  system  communicating  internally  to  co-ordinate  its  overall  activity. 

As  indicated  earlier,  the  System  Model  in  EMD  represents  the  system  as  a  set  of  co-operating  autonomous 
agents.  Provisions  are  also  made  for  external  agents  to  play  a  role  supporting  the  goals  of  both  human  users 
and  system  agents. 

5.6.2.10  Hierarchical  Goal  Analysis  (HGA)  of  Tasks  in  the  Help  System 

Hierarchical  goal  analysis  (see  above,  Section  5.4  Perceptual  Control  Theory,  for  a  discussion  of  HGA)  can  be 
applied  to  the  task  goal  hierarchy  for  the  help  system.  In  the  same  way,  HGA  can  also  be  used  in  an  analysis 
and  design  of  a  goal  hierarchy  for  a  network  of  intelligent  agents  to  assist  human  users.  That  analysis  offers 
the  same  benefits  described  earlier:  a  thorough  decomposition  of  the  goal-plan-action  hierarchy  along  with 
stability  and  information  flow  analyses. 

The  highest-level  agent  network  goal  could  take  the  general  form:  “to  perceive  (believe)  that  the  user  is 
performing  the  current  task  sufficiently  well,”  i.e.,  that  the  user  does  not  require  assistance  from  any  system 
agent.  That  goal  then  can  be  decomposed  into  sub-goals  regarding  the  perception  of  different  signs  that  a  user 
needs  assistance,  e.g.,  “to  perceive  (believe)  that  the  user  is  performing  the  current  task  in  the  most  efficient 
manner.”  An  error  signal  would  result  when  inefficiencies  in  the  user’s  actions  are  detected,  leading  system 
agents  to  consider  intervention. 

5.6.2.11  Ecological  Interface  Design 

Ecological  Interface  Design  (EID)  is  a  framework  for  problem  domain  analysis  and  the  design  of  human- 
machine  interfaces  in  complex  work  environments.  The  approach  incorporates  elements  from  ecological 
psychology,  particularly  the  emphasis  on  the  importance  of  considering  the  interaction  of  humans  with  their 
environment.  While  most  traditional  interface  design  approaches  confine  their  attention  to  human 
characteristics,  EID  also  examines  how  humans  interact  with  their  surroundings,  taking  into  account  both 
physical  and  cognitive  factors,  in  the  context  of  the  complex  system  under  control.  To  achieve  that,  EID  offers 
concrete  guidelines  on  interface  design,  with  the  aim  of  producing  optimal  usability  and  safety.  See  [2]. 

EID  should  be  considered  for  use  in  the  context  of  applications  that  are  safety-critical  and  involve  high 
cognitive  workloads. 

5.6.2.12  Integrated  Methodology  for  Help  System  Design 

The  foregoing  sections  presented  a  variety  of  theoretical  approaches  to  construct  a  comprehensive,  integrated 
framework  for  the  design  and  implementation  of  an  intelligent,  adaptive,  agent-based  system  for  providing 
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help  to  software  users.  The  resulting  integrated  methodology  is  composed  of  elements  from  the  following 
design  approaches: 

•  CommonKADS  (CK)  -  a  knowledge  management  and  engineering  methodology  that  guides  the 
systematic  analysis  and  design  of  intelligent  systems; 

•  IDEF  Standards  -  a  complement  to  the  CommonKADS  methodology  through  its  more  effective 
support  for  temporal  modelling  and  ontology  construction; 

•  Explicit  Models  Design  (EMD)  -  a  methodology  for  building  models  that  identify  and 
compartmentalise  the  knowledge  required  by  intelligent  systems; 

•  Perceptual  Control  Theory  (PCT)  -  a  feedback  control  system  model  for  goal-directed  behaviour  in  a 
system;  and 

•  Software  Agent  Paradigm  -  a  software  design  approach  that  supports  enhanced  modularity, 
reusability  and  efficiency. 

The  integration  of  the  above  techniques  into  a  comprehensive,  cross-disciplinary  design  framework  serves  the 
goals  of  generating  a  robust,  maintainable  and  reliable  help  system. 

Following  is  a  description  of  the  recommended  procedure  for  designing  and  implementing  a  knowledge 
system  within  that  framework.  Steps  are  designed  to  be  pursued  in  the  sequence  presented. 

To  facilitate  the  presentation  of  the  procedure,  a  legend  is  provided  to  help  distinguish  the  models  that 
comprise  the  CommonKADS  (CK)  knowledge  and  engineering  methodology  and  those  used  in  Explicit 
Models  Design  (EMD). 


CommonKADS  (CK): 

Organisation  Model 
Task  Model  (CK) 
Agent  Model 
Knowledge  Model 
Communication  Model 


Explicit  Models  Design  (EMD): 

Task  Model  (EMD) 

User  Model 
System  Model 
Dialogue  Model 
World  Model 


Co-ordination  Model  (MAS-CommonKADS) 

Design  Model 

5.6.2.13  Help  System  Design  Methodology 

•  Construct  an  Organisation  Model  to  describe  the  organisational  structure  within  which  the  knowledge 
system  will  be  used; 

•  Construct  the  Task  Model  (CK),  including  task  hierarchies  for  all  agents  identified  above  (use  IDEF3  to 
represent  the  hierarchies  in  applications  with  precise  timing  needs); 

•  Construct  the  Agent  Model  identifying  all  user  and  system  agents  and  their  relationships; 

•  Generate  the  Task  Model  (EMD)  by  extending  the  Task  Model  (CK)  to  produce  task  hierarchies  for  all 
agents  using  PCT-based  hierarchical  goal  analysis; 
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•  Develop  the  User  Model  according  to  the  need  to  track  user  preferences  and  knowledge; 

•  Specify  the  content  of  the  System  Model  to  enable  representation  and  use  of  system  preferences  and 
knowledge; 

•  Design  the  World  Model  to  contain  required  information  about  the  environment  necessary  for  the 
knowledge  system  to  operate  effectively; 

•  Specify  the  Dialogue  Model,  Communication  Model  and  Co-ordination  Model  to  govern  the  format  and 
content  of  communication  among  agents  (ensure  that  the  ability  exists  for  agents  to  provide  feedback  to 
one  another); 

•  Use  IDEF5  to  design  an  ontology  to  represent  the  contents  of  all  Explicit  Models; 

•  Develop  the  Knowledge  Model  to  encapsulate  the  ontology  and  an  associated  knowledge  base  containing 
information  from  all  Explicit  Models; 

•  Within  the  Knowledge  Model,  represent  the  Task  Model  (EMD)  as  a  hierarchy  of  PCT  loops  that  use  plan 
recognition  and  plan  generation  to  form  input  perceptions  and  output  behaviours;  and 

•  Create  the  Design  Model  to  produce  design  specifications  for  the  target  knowledge-based  system. 

5. 6.2. 13.1  Generalised  Principles  of  Help  System  Design 

Now  that  the  theoretical  infrastructure  underlying  the  help  system  has  been  described,  it  is  useful  to  examine 

some  general  principles  for  guiding  the  construction  of  an  adaptive  interface: 

•  Combine  principles  from  CommonKADS,  IDEF  Standards,  Explicit  Models  Design,  Perceptual  Control 
Theory  and  agent-oriented  development  to  design  and  implement  the  system,  as  described  above; 

•  Apply  the  five-part  taxonomy  within  the  plan  recogniser  to  classify  user  interface  actions  in  terms  of  user 
help  requirements.  The  taxonomy  of  requirements  includes  efficiency,  completeness,  consistency, 
correctness  and  safety,  which  guide  the  establishment  of  a  network  of  rules  that  enable  the  system  to 
determine  a  user’s  help  requirements; 

•  Use  the  results  of  the  user  needs  analysis  to  develop  plans  that  provide  optimal  help  given  the  system’s 
on-going  knowledge  of  the  user; 

•  Structure  the  interface  to  maximise  the  effective  execution  of  tasks  with  respect  to  the  five  criteria  above; 

•  Structure  the  interface  to  ensure  that  users  receive  continuous  support  in  executing  plans  and  achieving 
goals.  A  variety  of  support  mechanisms  should  be  implemented,  including,  among  other  things,  wizards, 
tutorials  and  ongoing  dialogues  between  the  user  and  system.  Their  purpose  is  to  help  create  a  virtual 
environment  where  help  is  integrated,  seamless  and  natural.  Not  only  should  those  help  mechanisms 
provide  guidance  and  education  on  software  use,  but  they  also  should  structure  the  user’s  pursuit  of  tasks 
in  a  way  that  allows  the  system  to  track  user  goals  and  plans  with  a  substantial  degree  of  accuracy. 
The  goal  is  for  the  system  to  take  maximum  advantage  of  opportunities  to  assist  users  while  minimising 
unnecessary  disruptions; 

•  Implement  each  object  in  the  software  as  an  intelligent  agent  responsible  for  monitoring  its  status  relative 
to  the  five  help  criteria; 

•  Allow  users  to  specify  a  “contract”  with  the  system  governing  the  nature  of  help  to  be  provided;  and 

•  Ensure  that  help  dialogue  encourages  bidirectional  feedback  between  the  user  and  the  system  sufficient  to 
mutual  understanding. 
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Those  items  are  described  in  greater  detail  in  the  following  sections. 

5.6.2.14  The  Five-Part  Taxonomy  for  Plan  Recognition 

In  the  “Perceptual  Control  Theory”  section,  a  set  of  five  criteria  was  introduced  for  use  in  classifying  whether 
the  user  is  carrying  out  the  current  task: 

•  Correctly; 

•  Completely; 

•  Consistently; 

•  Efficiently;  and 

•  Safely. 

The  control  loops  monitoring  user  activity  should  implement  rules  to  determine  whether  the  above  criteria  are 
being  met.  When  a  failure  is  identified,  the  system  is  signalled  (by  one  or  more  agents)  that  the  user  may 
require  help.  A  key  advantage  of  that  approach  is  that  the  rules  are  not  application-specific  and  can  be 
implemented  in  any  help  system  that  includes,  or  can  construct,  a  complete  hierarchy  of  tasks  and  plans  for 
the  application.  Additionally,  most  of  the  help  information  presented  to  users  as  a  result  of  the  rules  can  be 
derived  directly  from  the  task  hierarchy  (e.g.,  showing  a  correct  sequence  of  actions  to  achieve  a  specific 
goal),  greatly  reducing  the  need  to  generate  custom  help  content. 

Following  is  a  discussion  of  the  rules  that  will  be  associated  with  each  of  the  five  criteria. 

5. 6. 2.14.1  Correctness 

Correct  execution  of  plans  is  associated  with  carrying  out  the  necessary  steps  in  the  required  order.  There  are  a 
few  possible  scenarios  when  the  system  observes  a  user  perform  an  action  that  is  not  the  expected  next  step 
in  the  currently  inferred  plan.  For  example,  it  is  entirely  possible  that  a  user  may  not  know  what  step  should 
be  performed  next  and  has  chosen  an  incorrect  action.  A  user  also  might  repeat  a  step  in  a  plan, 
either  accidentally  or  because  of  a  misunderstanding  about  the  correct  procedure.  Such  scenarios  provide 
opportunities  for  the  system  to  offer  clarification  on  the  correct  approaches. 

Alternatively,  a  user  may  have  abandoned  the  current  plan,  which  again  offers  a  possible  help  opportunity, 
since  the  abandonment  may  be  the  result  of  that  user  not  knowing  the  correct  steps  required  to  finish  what  was 
begun. 

There  is  also  the  possibility  that  a  user  has  deliberately  decided  not  to  pursue  the  original  plan,  or  to  follow  an 
alternate  plan  simultaneously.  The  knowledge  in  the  User  Model  relating  to  the  user’s  expertise,  knowledge  of 
particular  software  functions  and  past  behaviour  executing  tasks  should  be  organised  in  a  way  to  facilitate  the 
system’s  ability  to  discriminate  among  such  possibilities.  Failing  that,  it  may  be  necessary  for  the  system  to 
ask  for  clarification  on  the  user’s  intentions  or  maintain  a  level  of  uncertainty  until  a  clear  plan  sequence  can 
be  identified. 

To  accommodate  such  inference  capabilities,  the  plan  recogniser  must  allow  for  the  possibility  that  multiple 
plans  are  being  pursued  concurrently. 
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5. 6. 2.14.2  Completeness 

Incomplete  execution  of  plans  also  can  be  associated  with  a  variety  of  scenarios.  For  example,  a  user  may 
omit  an  action  while  executing  a  plan,  suggesting  the  system  should  inform  the  user  of  the  missed  step.  A  user 
also  might  stop  carrying  out  a  plan  before  all  the  steps  are  complete,  resulting  in  a  need  for  the  system  to 
identify  whether  abandoning  the  plan  was  intentional  or  inadvertent  by  examining  the  User  Model  and  task 
history,  and  possibly  asking  for  clarification. 

Delays  in  completing  a  plan  may  occur  when  users  are  distracted  or  are  pursuing  another  plan  at  the  same 
time.  In  such  cases,  it  may  be  desirable  to  present  a  reminder  about  the  unfinished  (suspended)  task. 

Some  actions  in  certain  plans  will  be  identified  as  optional  (either  in  their  ordering  or  in  their  presence  in  a 
plan)  and  completeness  rules  will  need  to  discern  whether  the  omission  of  an  optional  step  violates  the 
completeness  criterion. 

It  should  be  noted  that  in  many  cases  there  is  overlap  between  the  completeness  and  correctness  criteria, 
where,  for  example,  if  a  user  skips  a  step  in  carrying  out  a  plan,  it  will  be  both  incorrect  and  incomplete. 
However,  beginning  a  plan  correctly  but  not  finishing  it  would  lead  to  a  correct,  but  incomplete  plan. 
In  contrast,  finishing  a  plan  with  an  incorrectly  executed  step  might  be  thought  of  as  a  “complete,” 
but  “incorrect”  plan.  Such  distinctions  justify  treating  the  two  criteria  separately. 

5. 6.2. 14.3  Consistency 

User  consistency  with  task  execution  can  be  determined  by  comparing  steps  taken  to  achieve  a  particular  goal 
with  those  taken  by  the  user  to  satisfy  that  goal  in  the  past.  Differences  can  reveal  a  number  of  different 
things.  Inconsistencies  in  the  performance  of  tasks  may  indicate  confusion  on  the  part  of  a  user  as  to  the 
correct  procedure,  presenting  an  opportunity  for  the  system  to  provide  help.  Inconsistencies  also  may  reveal 
that  a  user  has  learned  a  new  plan  for  achieving  a  goal,  and  that  new  knowledge  should  be  noted  in  the  User 
Model. 

An  important  point  is  that  an  observation  by  the  system  that  a  user  is  performing  a  task  inconsistently  with  his 
or  her  past  behaviour  may  not  indicate  that  intervention  is  required,  unless  there  are  signs  of  incorrect  or 
inconsistent  plan  execution. 

A  user  who  varies  his  methods  of  carrying  out  plans  may  simply  be  trying  to  achieve  variety  in  their 
execution.  Knowledge  of  such  tendencies  should  be  stored  in  the  User  Model  to  facilitate  future  classification 
of  user  activity  according  to  the  five  criteria. 

Establishing  consistency  requires  knowledge  of  past  activity  and  the  system  will  not  have  that  historical  data 
for  users  who  are  new  to  the  help  system.  Because  inexperienced  users  likely  will  require  immediate  help, 
default  settings  for  typical  users  will  serve  as  a  starting  point  for  building  a  user  profile.  For  intermediate  and 
advanced  users,  however,  the  system  may  conduct  a  period  of  observation  to  become  familiar  with  that  user’s 
preferences,  needs  and  methods. 

Maintenance  of  activity  profiles  should  be  an  ongoing  process  for  both  novice  and  expert  users.  The  system 
should  analyse  history  information  contained  in  the  Task  Model  for  a  user  to  determine  preferred  plans  for 
accomplishing  goals  and  knowledge  of  software  features.  By  performing  such  analyses,  historical  task  data 
can  be  archived  more  concisely  and  accessed  more  quickly. 
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5.6.2.14.4  Efficiency 

A  simple  rule  for  identifying  inefficiency  in  user  actions  would  compare  the  current  plan  with  alternate  plans 
for  achieving  the  same  goal.  The  presence  of  another  plan  with  fewer  steps  suggests  that  the  user  should  be 
informed  of  the  simpler  alternative. 

There  may  be  situations  where  alternate  plans  require  the  same  number  of  steps,  but  where  one  would  be 
considered  more  efficient.  For  example,  selecting  a  menu  item  using  a  keyboard  shortcut  is  often  faster  than 
using  the  mouse  to  pull  down  the  menu.  Also,  some  keyboard  shortcuts  are  easier  to  perform  than  others, 
such  as  pressing  the  Delete  key  instead  of  Ctrl-X  to  delete  an  object.  A  system  of  rules  should  take  such 
factors  into  account  and  assess  the  relative  ease  with  which  competing  plans  could  be  executed. 

Another  issue  is  that  a  particular  action  may  be  more  or  less  efficient  depending  on  the  other  actions  that  are 
part  of  the  same  plan.  For  example,  if  a  user  has  been  typing  data  into  a  dialogue  box,  it  typically  will  be  most 
efficient  to  press  the  Return  key  to  dismiss  the  window.  However,  in  circumstances  where  the  user’s  hand  is 
already  on  the  mouse,  it  would  be  easier  to  click  the  OK  button.  Rules  in  the  knowledge  system  should 
evaluate  the  efficiency  of  steps  in  the  context  of  the  broader  plan. 

A  final  rule-based  method  to  increase  user  efficiency  involves  identifying  plans  that  the  user  is  pursuing  that 
could  be  completed  by  the  system  without  further  input  from  the  user.  In  such  situations,  the  system  either 
could  automatically  complete  the  task  for  the  user,  or  could  offer  to  do  so  with  the  user’s  approval,  depending 
on  the  selected  automation  level  in  the  user’s  contract  with  the  system.  In  order  to  offer  such  capabilities, 
the  help  system  should  be  able  to  distinguish  between  actions  in  the  goal  and  plan  hierarchy  that  require 
human  input  and  those  that  do  not.  An  example  of  a  task  requiring  no  human  input  is  the  ability  of  most  web 
browser  software  to  enter  information  into  online  forms  automatically. 

5.6.2.14.5  Safety 

Safe  execution  of  tasks  will  not  be  a  critical  issue  in  all  applications,  but  in  some  domains  (e.g.,  aviation, 
industrial  process  control),  the  safety  of  humans  and  equipment  can  be  a  deciding  factor. 

In  practically  all  software  applications  there  is  a  safety  issue  surrounding  the  avoidance  of  data  loss.  A  simple 
example  of  a  safety  mechanism  is  the  standard  prompt  to  save  changes  when  a  file  is  closed,  but  more 
involved  methods  can  be  imagined.  For  example,  new  users  could  be  presented  with  additional  warnings  when 
deleting  complex  objects,  reverting  to  default  settings  or  carrying  out  other  tasks  where  a  substantial  amount 
of  information  could  be  erased  inadvertently.  The  availability  of  a  “multiple  undo”  capability  helps  reduce  the 
risk  of  irreversible  violations  of  safety  criteria. 


5.6.2.15  The  Five-Part  Taxonomy  for  Plan  Generation 

The  on-going  user-needs  analysis,  described  in  the  preceding  section,  is  responsible  for  determining  whether 
or  not  a  user  requires  help  with  respect  to  each  of  the  five  help  criteria.  Deciding  when  and  how  best  to 
provide  that  help  falls  to  plan  generation  techniques. 

Generated  plans  will  depend  on  the  nature  of  a  user’s  need  for  help.  Following  is  a  description  of  help  needs 
as  they  fall  within  the  five-part  taxonomy. 
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5. 6. 2.15.1  Correctness 

If  a  plan  is  being  pursued  incorrectly,  it  is  likely  that  the  user  should  be  informed  about  it  promptly. 
For  example,  if  an  incorrect  step  is  performed  in  a  plan,  the  user  will  need  to  know  that  his  goal  may  not  have 
been  satisfied  since  the  plan  was  executed  incorrectly.  Thus,  the  correct  procedure  for  accomplishing  that  goal 
should  be  conveyed  to  the  user  promptly  and  directly,  perhaps  through  the  use  of  an  immediate  help  window. 

On  the  other  hand,  when  there  is  system  ambiguity  as  to  whether  a  user  is  carrying  out  a  plan  incorrectly, 
or  has  simply  abandoned  it  altogether,  it  will  be  useful  to  present  that  user  with  a  question  about  her 
intentions.  The  help  system  should  allow  clarification  questions  to  be  presented  to  users  in  a  non-disruptive 
fashion,  such  as  through  the  use  of  a  floating  window.  Although  responding  to  such  questions  will  be  at  a 
user’s  discretion,  doing  so  will  improve  the  help  system’s  model  of  that  user,  and  therefore  its  ability  to 
provide  useful  help  to  her. 

A  user  may  wish  to  monitor  plans  that  the  system  believes  she  is  pursuing  and  that  could  inform  her  that  an 
action  is  not  a  correct  part  of  the  current  plan.  That  monitoring  could  occur  in  a  window  showing  the  currently 
inferred  plan  in  the  context  of  higher-level  tasks,  as  well  as  what  the  next  action  should  be.  It  also  could  alert 
the  user  that  the  system  believes  she  is  pursuing  some  goal  other  than  her  true  goal.  A  mechanism  could  be 
provided  for  the  user  to  correct  the  system’s  misunderstanding  by  specifying  the  actual  plan  being  pursued, 
which  will  provide  a  learning  opportunity  for  the  system.  While  most  users  will  not  find  it  practical  to  monitor 
such  a  display  continuously,  it  will  be  very  useful  in  situations  where  users  are  unsure  of  correct  procedures. 
Providing  a  monitoring  mechanism  for  a  user  could  provide  opportunities  to  understand  how  such  information 
might  be  organised  and  presented  more  effectively  in  future.  It  could  serve  as  an  experimental  design  element 
with  some  option  for  user  feedback. 

5. 6. 2.15.2  Completeness 

An  incomplete  plan  can  be  the  result  of  a  user  omitting  a  required  action  during  its  execution  and,  as  such, 
it  is  important  to  inform  the  user  of  that  omission.  In  those  cases,  a  help  window  should  be  displayed 
immediately  so  that  the  user  may  correct  the  error  and  achieve  the  intended  goal. 

Incompleteness  may  also  result  from  abandoning  a  plan  and  pursuing  another  in  its  place.  In  such  cases, 
it  may  be  important  to  establish  through  a  clarification  dialogue  whether  the  abandonment  was  the  result  of  a 
shift  in  intentions  or  if  it  arose  because  the  user  lacked  required  knowledge.  The  latter  case  would  signal  a 
need  to  present  the  missing  knowledge.  A  user  might  also  abandon  a  plan  with  which  he  is  unfamiliar  in 
favour  of  one  that  he  is  confident  will  work.  In  that  case,  and  under  the  right  circumstances,  the  system  should 
offer  assistance  on  the  initial  plan  about  which  he  was  unsure.  Situations  where  users  are  pursuing  multiple 
simultaneous  plans  or  have  been  distracted  would  not  warrant  intervention  from  the  system,  but  provide  an 
opportunity  for  clarification.  Questions  seeking  clarification  should  be  presented  unobtrusively,  since  it  will 
not  always  be  possible  for  a  user  to  suspend  the  current  task  to  respond  to  the  help  system.  Clarification 
questions  may  be  presented  in,  for  example,  a  help  system  status  line,  to  enable  users  to  monitor  system 
activity.  In  situations  where  the  user  fails  to  notice  the  question  within  a  reasonable  period  of  time,  the  system 
should  alert  him  to  its  presence.  Care  must  be  taken  to  make  such  alerts  noticeable  but  not  intrusive  and, 
where  possible,  context  should  be  considered  before  displaying  the  alert.  In  applications  where  space  is 
insufficient  to  present  questions  in  full,  a  discreet  notification  may  be  used,  such  as  the  display  of  an  icon. 
The  “Plans  for  Providing  Help”  section,  below,  offers  further  discussion  of  methods  for  presenting 
clarification  questions. 


RTO-TR-HFM-078 


5-63 


ARTIFICIAL  COGNITION  AND  CO-OPERATIVE  AUTOMATION 


5. 6. 2.15.3  Consistency 

Some  inconsistent  behaviour  on  the  part  of  a  user  may  indicate  confusion  as  to  how  to  achieve  current  goals. 
Unlike  correctness  and  completeness,  the  consistency  criterion  often  will  not  be  associated  with  an  overt  sign 
that  the  user  requires  assistance  and  so,  typically,  will  not  call  for  the  same  immediate  presentation  of  help 
information.  In  those  cases,  it  will  be  more  appropriate  to  use  a  less  overt  form  of  help,  such  as  waiting  for  a 
pause  in  a  user’s  activities  before  presenting  information,  or  displaying  a  subtle  prompt  to  tell  the  user  help  is 
available. 

5.6.2.15.4  Efficiency 

As  with  consistency,  the  efficiency  criterion  usually  is  not  associated  with  problems  in  completing  the  current 
task  and,  therefore,  an  inconspicuous  or  delayed  form  of  help  is  also  appropriate.  Delayed  help  will  presented 
upon  completion  of  the  current  plan,  at  idle  time,  at  the  end  of  the  session  or  under  other  circumstances  that 
will  not  disrupt  user  performance. 

5.6.2.15.5  Safety 

The  safe  execution  of  tasks  is  associated  with  a  high  priority  for  conveying  help  to  the  user.  Unsafe  acts  may 
or  may  not  involve  risk  to  human  safety,  but  there  may  be  risks  of  data  loss.  Help  information  relating  to 
safety  will  necessitate  immediate  notification  of  the  user  in  a  conspicuous  manner,  such  as  with  an  auditory 
signal  accompanying  a  dialogue  box  requiring  user  acknowledgement. 

When  accidents  occur  in  safety-critical  domains,  such  as  aviation,  they  are  typically  followed  by 
investigations  to  identify  their  attendant  causes.  An  accident  occurring  in  the  context  of  a  user  interacting  with 
an  intelligent  system  provides  an  opportunity  for  investigation  results  to  assist  software  developers  in 
redesigning  systems  to  prevent  future  accidents.  The  software  application,  Systematic  Error  and  Risk  Analysis 
(SERA)  Tool,  was  developed  to  help  accident  investigators  identify  failures  and  their  pre-conditions  using 
principles  of  Perceptual  Control  Theory.  Results  of  investigations  using  SERA  would  be  useful  to  designers 
seeking  to  build  safer  software  systems. 

5.6.2.16  Plans  for  Providing  Help 

Help  can  be  offered  to  users  in  a  variety  of  forms: 

•  A  “wizard”  interface  to  guide  the  user  through  a  complex  process,  such  as  creating  a  workspace  and 
configuring  its  contents  (see  the  next  section  on  “Wizards”). 

•  Tutorials  tailored  to  a  user  according  to  what  the  system  believes  the  user  knows.  It  may  be  presented 
according  to  a  prearranged  schedule  resulting  from  an  agreement  (PACT  “contract”  -  see  “The  PACT 
Approach  and  Automation  Levels”  section,  below)  between  the  user  and  system  (e.g.,  a  weekly  mini¬ 
tutorial  on  a  feature  with  which  the  user  is  not  familiar). 

•  Interactive  tutorials,  whereby  the  system  steps  through  a  description  of  a  procedure  while  the  user 
carries  it  out. 

•  Tutorials  providing  guidance  on  how  to  solve  an  application-specific  problem  that  a  user  is 
confronting.  That  would  occur,  say,  in  response  to  a  user  asking,  “How  do  I  finish  this  [the  task  the 
user  has  begun]?”. 

•  Presentation  of  a  question  asking  if  the  user  would  like  assistance  or  a  question  asking  for 
clarification  of  a  user’s  intentions: 
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Such  questions  could  be  displayed  in  a  help  system  panel  within  the  main  application  window, 
and  could  be  accompanied  by  one  or  more  indicators,  such  as  light  bulb  icons,  to  show  the  availability 
of  messages  from  the  help  system.  A  small  panel  could  show  the  current  question  or  partial  text  of 
that  question  so  a  user  could  decide  whether  to  respond.  If  the  user  did  not  respond  within  a  specified 
time,  the  light  bulb  icon  and  question  would  flash  two  or  three  times  to  attract  the  user’s  attention. 
The  question  would  disappear  if  it  continued  to  be  ignored,  perhaps  after  a  second  round  in  which  the 
light  bulb  flashed.  Users  still  should  have  the  option  of  reviewing  that  material  at  their  convenience, 
at  the  end  of  a  session  or  in  other  circumstances  deemed  appropriate;  and 

If  the  system  observed  a  prolonged  period  without  user  activity,  it  might  reasonably  conclude  that  the 
user  had  suspended  activity  and  was  away  from  the  computer.  In  such  a  case,  the  presentation  and 
other  system-related  activities  with  respect  to  help  signals  to  the  user  would  be  delayed  until  work 
with  the  software  resumed. 

•  During  any  review  of  system  proffered  help,  users  would  be  able  to  examine  the  context  in  which  it 
was  determined  that  help  was  appropriate,  specifically  the  context  of  attendant  user  actions  and 
system- inferred  goals  and  plans  from  those  actions.  The  display  of  questions  and  optional  review  will 
be  controlled  as  part  of  the  user’s  PACT  agreement  with  the  system  (see  “The  PACT  Approach  and 
Automation  Levels”  section,  below). 

•  Presentation  of  web-based  help. 

•  Display  of  a  light  bulb  or  other  alert  indicating  the  availability  of  help  information  when  selected. 

•  Brief  pop-up  descriptions  of  interface  elements  when  the  mouse  is  “over  top”. 

•  An  offer  to  complete  a  task  that  the  user  has  started. 

•  Periodic  presentation  of  tips  with  which  the  system  believes  the  user  is  unfamiliar. 

5. 6. 2.16.1  Wizards 

To  further  the  goal  of  creating  an  interface  in  which  users  receive  continuous  support  in  executing  plans  and 
achieving  goals,  a  variety  of  wizards  should  be  available  to  users.  The  wizard  interface  approach  offers  a 
number  of  advantages: 

•  It  simplifies  tasks  for  users  because  the  system  takes  care  of  detailed  plans  and  contextual  supports 
along  the  way,  making  it  useful  for  both  novice  and  expert  users; 

•  Incidental  learning  occurs  when  users  are  guided  through  procedures  for  achieving  high-level  goals; 

•  By  stepping  a  user  through  a  well-defined  procedure,  the  help  system  can  infer  the  user’s  intentions 
with  much  greater  confidence  (especially  true  for  novice  users)  than  if  he  were  proceeding  on  his  own 
initiative.  That  substantially  increases  the  relevance  and  usefulness  of  help  offered  by  the  system 
about  the  task;  and 

•  Given  the  built-in  constraints  on  task  execution,  there  will  be  fewer  opportunities  for  plans  to  violate 
any  of  the  help  system  criteria. 

To  help  design  and  implement  a  wizard,  scenarios  will  be  developed  to  describe  how  typical  knowledge 
elicitation  sessions  proceed.  That  information  will  be  useful  in  establishing  specifications  for  audio  wizards, 
described  next. 
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5.6.2.16.1.1  Audio  Wizards 

Consideration  should  be  given  to  the  creation  of  a  wizard  interface  that  uses  an  audio  dialogue  between  the 
user  and  system.  Reliable  speech  tools  are  now  widely  available  and  the  incorporation  of  those  techniques  into 
a  help  system  is  a  logical  way  to  make  dialogue  more  natural  between  the  system  and  user. 

In  an  audio  wizard,  questions  are  posed  using  software  speech  generation  and  spoken  answers  from  users  are 
interpreted  using  speech  recognition.  Questions  should  be  phrased  to  limit  the  range  of  possible  responses  to 
ensure  high  recognition  accuracy. 

5.6.2.17  Control  Loop  Design 

5. 6.2. 17.1  Centralised  System  Model  Control  Loops 

Responsibilities  for  determining  user  help  needs  are  divided  among  the  System  Model  and  agents  associated 
with  individual  interface  objects.  Control  loops  in  the  System  Model  should  monitor  user  actions  and  apply 
the  general  rules  identified  in  the  earlier  section,  “The  Five-Part  Taxonomy  for  Plan  Recognition.” 
That  includes  a  hierarchy  of  rules  based  on  those  introduced  in  the  “Perceptual  Control  Theory”  section. 
A  high-level  description  of  control  loop  goals  is  as  follows: 

To  perceive  (believe)  that. . . 

•  . .  .the  user  is  performing  the  current  task  sufficiently  well  and  does  not  require  help; 

•  . .  .the  user  is  performing  the  current  task  efficiently; 

•  . .  .there  are  no  plans  with  fewer  steps  available  to  achieve  the  current  goal; 

•  ...there  are  no  more  efficient  plans  of  the  same  length  to  achieve  that  goal  (e.g.,  shorter  time 
requirements  or  greater  convenience  for  the  user); 

•  . .  .the  user  is  performing  the  task  correctly; 

•  . .  .the  user  has  not  performed  a  step  that  is  not  in  the  current  plan; 

•  . .  .the  user  has  not  performed  steps  in  the  current  plan  that  are  out  of  order; 

•  . .  .the  user  has  not  repeated  a  step  in  the  current  plan  unnecessarily; 

•  . .  .the  user  is  performing  the  task  completely; 

•  . .  .the  user  has  not  omitted  one  or  more  steps  in  the  current  plan; 

•  . .  .the  user  has  not  suspended  the  pursuit  of  a  plan; 

•  . .  .the  user  is  performing  the  task  consistently; 

•  . .  .the  user  has  not  behaved  in  a  manner  inconsistent  with  past  task  performance; 

•  . .  .the  user  is  performing  the  task  safely;  and 

•  ...the  user  is  performing  the  task  consistent  with  the  integrity  of  key  human,  machine  and  data 
elements  of  the  total  system. 

5.6.2.18  Objects  as  Intelligent  Agents 

All  interface  objects  in  the  target  software  should  be  implemented  as  a  network  of  intelligent  agents, 
which  has  the  effect  of  increasing  modularity,  organisation  and  reusability  among  help  system  components. 
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(See  the  “Software  Agent  Paradigm”  section) 

Agent-based  objects  are  to  include  all  standard  interface  elements,  such  as  windows,  buttons  and  menu  items, 
as  well  as  application-specific  objects.  Each  agent  is  responsible  for  monitoring  its  own  status  relative  to  the 
five  help  criteria,  described  earlier.  Those  include  checking  the  following  properties: 

•  Correctness  -  e.g.,  that  the  value  of  a  variable  or  the  contents  of  a  text  box  is  within  an  acceptable 
range; 

•  Completeness  -  e.g.,  that  all  data  have  been  entered  for  a  particular  object; 

•  Consistency  -  e.g.,  that  data  are  consistent  with  the  execution  of  current  tasks; 

•  Efficiency  -  e.g.,  that  a  frequently  used  menu  item  is  associated  with  an  easily  accessed  keyboard 
shortcut;  and 

•  Safety  -  e.g.,  that  some  agents  will  have  tendencies  for  self-preservation,  particularly  where  loss  of 
data  is  a  risk. 

Agent  processes  monitoring  those  criteria  should  be  continuous  and  System  and  User  Models  should  be  able 
to  query  agents  as  needed. 

Agents  should  monitor  the  usage  of  their  associated  objects  by  the  current  user  and  store  important 
information,  such  as  when  they  were  first  accessed,  the  most  recent  access  and  the  number  of  times  they  have 
been  accessed.  That  information  will  be  available  for  compiling  User  Model  knowledge  about  software 
features  and  user  familiarity  with  them. 

As  noted  earlier  in  the  “Centralised  System  Model  Control  Loops”  section,  the  System  Model  should  monitor 
user  needs  in  the  context  of  the  general  rules  of  the  five-part  taxonomy.  Specific  rules  should  be  associated 
with  interface  objects  and  their  agents. 


5.6.2.19  The  PACT  Approach  and  Automation  Levels 

Since  it  is  unrealistic  to  expect  that  all  users  will  require  the  same  level  of  automation  from  an  intelligent 
system  at  all  times,  a  need  exists  for  users  to  be  able  to  specify  their  requirements  of  the  system.  In  the  help 
system,  circumstances  of  a  particular  task  and  operator  preferences  will  dictate  what  automation  level  is  most 
appropriate,  and  those  could  change  over  the  course  of  a  session. 

One  method  of  handling  automation  levels  in  the  air  domain  is  the  Pilot  Authorisation  and  Control  of  Tasks 
(PACT)  system.  That  approach  is  based  on  the  notion  of  contractual  autonomy,  in  which  a  user  and  system 
establish  an  agreement,  or  contract,  on  the  system’s  responsibilities.  Contracts  are  made  using  a  system  of  six 
levels,  numbered  from  0  (no  automation)  to  5  (fully  automatic).  Table  5-4  shows  the  levels  of  autonomy  in  the 
PACT  approach. 
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Table  5-4:  PACT  Levels  of  Autonomy 


Levels 

Operational 

Relationship 

Computer  Autonomy 

Pilot  Authority 

5 

Automatic 

Full 

Interrupt 

4 

Direct  Support 

Advised  action  unless 
revoked 

Revoking  action 

3 

In  Support 

Advice,  and  if  authorised, 
action 

Acceptance  of  advice  and 
authorising  action 

2 

Advisory 

Advice 

Acceptance  of  advice 

1 

At  Call 

Advice  only  if  requested 

Full  pilot,  assisted  by  computer 
only  when  requested 

0 

Under  Command 

None 

Full 

Contracts  in  a  help  system  should  offer  users  the  ability  to  set  the  autonomy  level  and  change  it  at  any  time 
during  a  session,  as  well  as  provide  the  flexibility  to  customise  the  provision  of  specific  forms  of  assistance. 
That  customisation  should  allow  users  to  request  help  at  specified  intervals  or  to  ask  that  help  be  provided  in  a 
specified  form. 

In  order  to  aid  in  deciding  the  most  appropriate  level  of  assistance  to  offer  a  user  given  the  selected  autonomy 
level,  the  system  should  maintain  a  set  of  numerical  scores  in  the  User  Model  to  indicate  the  preferences  and 
needs  of  the  current  user.  Those  act  as  thresholds  in  determining  when  the  system  should  intervene,  based  on: 

•  System  beliefs  that  the  user  possesses  relevant  knowledge; 

•  Feedback  from  the  user  in  response  to  each  help  offer  from  the  system,  either  by  using  a  “Don’t  show 
again”  button  or  by  ranking  the  usefulness  of  the  information  on  a  scale  of  1  to  10,  as  in,  “How  would 
you  rate  the  value  of  this  help?”  The  system  also  should  have  the  ability,  at  least  in  some 
circumstances,  to  judge  the  suitability  of  displaying  a  message  again,  even  without  a  user  selecting 
the  “Don’t  show  again”  option.  In  most  cases,  after  information  has  been  displayed  and 
acknowledged  by  the  user,  the  system  will  infer  that  knowledge  will  not  need  to  be  presented  again, 
unless  subsequent  user  actions  demonstrate  that  it  was  not  understood  or  has  been  forgotten; 

•  General  feedback  from  the  user  on  preferred  types  of  help,  based  on  online  questionnaire  responses 
from  the  system  on  favoured  techniques.  Such  a  questionnaire  could  elicit  general  information  on  the 
efficacy  of  help  provided  to  a  user,  perhaps  at  the  end  of  a  session.  The  PACT  contract  will  allow 
users  to  enable  or  disable  the  feature  according  to  their  desire  for  such  a  feature; 

•  Historical  system  knowledge  of  techniques  that  have  been  most  effective  at  conveying  information  to 
that  user  in  the  past,  revealed  by  acknowledged  help  messages  and  demonstrated  task  knowledge;  and 

•  State  of  completion  of  the  current  task,  so  as  to  avoid  unnecessary  disruption. 

5.6.2.20  References 
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5.6.3  Autonomous  Decision  Making  for  an  Underwater  Unmanned  Vehicle 

5.6.3. 1  Underwater  Unmanned  Vehicles 

Unmanned  vehicles  have  been  around  for  some  years  in  all  environments  -  on  the  ground  (UGV),  in  the  air 
(UAV),  surface  water  (USV)  and  underwater  (UUV).  More  strictly,  these  have  been  primarily  in  the  category 
of  remotely  operated  vehicles  (ROV),  in  that  there  are  command  and  communications  links  (direct  or  via 
satellite)  to  a  remote  human  operator,  who  maintains  full  system  control  at  all  times. 

In  the  UAV  field  in  particular,  the  effort  is  directed  at  improved  data  fusion  and  representation,  in  order  to 
allow  the  human  operator  to  control  several  UAVs  at  once,  furnishing  the  vehicles  with  autonomy  over  lower 
level  functionality,  but  nowhere  near  full  autonomy.  Underwater  ROVs  tend  to  be  tethered  to  a  mother  ship 
via  an  umbilical  cord,  supplying  power  and  command  and  communications  links  [1]. 

The  free  swimming  UUVs  tend  to  be  very  small,  of  limited  endurance,  with  a  single  specific  task,  such  as 
inspection  of  underwater  objects  [2].  In  all  these  cases  the  man  is  being  kept  firmly  in  the  loop. 

However,  the  envisaged  operational  environment  for  military  UUVs  precludes  such  an  umbilical  link  and, 
indeed,  for  certain  mission  phases,  any  sort  of  communications  with  the  vehicle  at  all.  This  will  require  the 
UUV  to  possess  the  capabilities  to  perform  all  the  tasks  to  be  performed,  from  navigation,  power 
monitoring/management,  threat  identification  and  avoidance  and  payload  delivery,  through  to  the  far  less 
concrete  area  of  high  level  decision  making  in  the  face  of  high  levels  of  uncertainty. 

The  power  source  will  also  have  to  be  wholly  internal  to  the  vehicle.  This  is  currently  based  on  batteries, 
although  it  is  likely  that  fuel  cells  will  replace  these  as  the  technology  improves.  Even  the  latter  will  only  yield 
a  useful  energy  output  of  about  400  Wh  per  kg,  with  the  potential  to  (possibly)  double  this  figure  within  the 
next  5  years.  With  current  UUVs,  such  as  the  USS  Manta,  requiring  up  to  50  kW  for  propulsion  alone,  plus 
several  kW  more  for  sensor  operation  (e.g.,  sonar),  the  size  of  the  problem  of  supplying  sufficient  power  to 
allow  the  performance  of  any  kind  of  mission  becomes  apparent. 

The  addressing  of  this  power  management  problem,  which  is  usually  denoted  by  HOTEL,  What  does  HOTEL 
mean?  It  boils  down  to  answering  the  question  6 can  I  do  the  mission  and  return  to  my  recovery  point  on  my 
power  reserves?’.  This  baselining  of  the  projected  energy  consumption  for  the  whole  mission,  continually 
updated  during  the  mission,  underpins  every  other  assessment  and  decision  made  during  the  mission. 

5.6.3.2  Envisaged  Theatre  of  Operations  for  Military  UUVs 

The  US  Navy  envisage  the  littoral  zone  to  be  the  most  important  for  UUV  operations  [3],  from  mine  counter 
measures  prior  to  a  naval  assault,  through  coastal  and  channel  mapping  via  sonar,  to  deploying  sonars  near 
enemy  naval  installations  to  track  asset  movement  and  even  kill  them  with  torpedoes.  This  poses  certain 
problems,  from  vehicle  design  through  to  sensor  operation.  The  traditional  teardrop  shape  is  the  most 
efficient,  from  a  drag  minimisation  point  of  view.  However,  this  is  not  appropriate  for  very  shallow  waters, 
giving  rise  to  a  significant  risk  of  running  aground.  Here,  a  thinner,  flatter,  more  Ray-like  profile  is  more 
appropriate  (e.g.,  Manta),  with  a  higher  power  consumption  as  a  result.  Conventional  sonars  will  have  their 
performance  degraded  significantly  from  bottom  clutter  in  the  shallows. 
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There  are,  however,  research  efforts  directed  at  biomimetic  sensors,  emulating  the  capabilities  of  the  sonar 
systems  of  dolphins,  which  do  not  suffer  significantly  from  such  problems.  This  will  be  crucial,  as,  in  all  these 
projected  scenarios,  the  UUV  will  be  totally  reliant  on  a  main  forward  sonar  and  left  and  right  short  range 
sidescan  sonars  to  prevent  it  from  running  aground  (or  into  obstacles),  as  it  follows  the  coastline  or  navigates 
through  narrow  channels. 

5.6.3.3  Approach  to  Automation  and  Decision  Making 

Whenever  we  automate  any  process,  there  are  certain  risks  associated  with  that  automation.  Obviously, 
in  some  cases  these  risks  are  lower  than  in  the  equivalent  manual  process,  in  others  they  are  higher.  The  3  risk 
factors  of  primary  concern  here  are  those  relating  to  communication  (right  information  at  the  right  time), 
workload  (system  overload)  and  unpredictability  (moving  outside  the  zone  covered  by  prior  knowledge  and 
experience). 

Figure  5.31  provides  a  schematic  representation  of  the  change  in  adaptiveness  vs.  levels  of  human  control, 
automatic  control  and  cognitive  cooperation,  for  these  3  important  risk  factors  of  communication,  workload 
and  unpredictability.  This  displays,  in  particular,  the  greater  susceptibility  of  human  operators  to  increased 
workload  vs.  an  automated  system  and  the  reverse  in  the  case  of  increased  uncertainty.  Any  ways  in  which  we 
can  improve  the  way  an  automated  system  can  handle  uncertainty  will  be  of  particular  benefit  in  the  context  of 
an  autonomous  system. 


Figure  5-31:  Cognitive  Co-operation  and  Human  vs.  Automation  Control. 


This  is  explored  further  in  Figure  5.32,  which  addresses  leveraging  autonomy  through  cognitive  automation. 
This  represents  the  opportunity  for  pushing  the  boundary  back  between  what  can  be  achieved  in  an  automated 
system  (traditionally  skill  and  rule  based  behaviour)  and  that,  knowledge  and  experience  based  area  that 
requires  human  control.  This  requires  the  ability  to  reason  effectively  in  the  presence  of  significant  levels  of 
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uncertainty.  This  can  be  based  on  a  combination  of  model  based  reasoning  and  a  suitable  methodology  for 
resolving  situations  in  which  no  clear  decision  is  forthcoming,  due  to  conflicting  objectives,  etc.  (e.g.,  neuro- 
fuzzy  or  genetic  algorithms,  e.g.,  [4]). 


Figure  5-32:  Leveraging  Autonomy  through  Cognitive  Automation. 


The  underlying  requirement  is  for  a  system  that  can  provide  for  dynamic,  context  sensitive  adaptiveness, 
so  as  to  engage  each  task  at  the  appropriate  level  of  autonomy. 

It  is  useful  here  to  simplify  the  different  (cognitive)  levels  of  tasks  into  3  groups,  namely  system  health 
monitoring,  mission  control  and  strategic  control.  Figure  5.33  shows  the  progression  of  the  dividing  line 
between  automated  control  and  human  control,  from  manned  underwater  vehicles,  through  the  traditionally 
understood  uninhabited  vehicle,  to  an  unattended  cognitive  underwater  vehicle,  which  is  an  intermediate  stage 
on  the  way  towards  our  goal  of  a  wholly  autonomous  (cognitive)  underwater  vehicle.  The  aim  is  to  steadily 
push  the  dotted  line  to  the  top  of  the  triangle,  through  improved  reasoning  and  uncertainty  handling  processes. 
The  causal  reasoning  process  can  be  represented  in  terms  of  a  decision  ladder,  consisting  of  a  clear  series  of 
knowledge  states,  linked  by  processes  which  determine  the  flow  between  these  states  (a  state  flow 
representation). 
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Figure  5-33:  Transition  from  Manned,  through  Current  Unmanned, 
to  Semi-Autonomous  Underwater  Vehicle. 

A  general  process  of  adaptiveness  is  shown  in  Figure  5.34.  Such  a  state  flow  representation  of  a  system  is, 
in  effect,  a  finite  state  machine  representation  of  it  and,  therefore,  by  definition  tractable.  It  can  be 
characterised  by  the  traditional  measures  from  information  theory  (Kolmogorov  Complexity  [9],  Fisher’s 
Information  Measure  [10],  Shannon’s  Negative  Entropy  [11],  etc.). 
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Figure  5-34:  Decision  Ladder  Framework  for  Task  Decomposition. 


In  particular,  the  aim  is  to  minimise  the  Kolmogorov  Complexity  of  the  system  representation.  In  effect, 
minimising  this  equates  to  an  Occam’s  Razor  approach  to  decision  making  and  problem  resolution.  This  is 
defined  as  the  shortest  algorithm  that  fully  represents  the  total  information  content  of  a  given  system. 
In  practice,  we  will  not  have  the  full  information  content  of  the  system  (i.e.,  mission)  and  will  be  operating  on 
a  best  approximation  to  that.  This  arises  directly  from  the,  at  times  considerable,  uncertainty  that  may  be 
present  during  the  course  of  a  mission  and  any  resultant  conflicts  within  the  decision  process. 

PACT  (Policy  for  Authorising  and  Control  of  Tasks)  provides  a  convenient  system  for  identifying  levels  of 
autonomy  in  a  system  (Commanded;  Assisted  -  At  Call,  Advisory,  In  Support,  Direct  Support;  Automated). 
The  system  is  one  that  has  been  applied  at  DERA  previously  for  addressing  decision  aiding  and  support  in  the 
Cognitive  Cockpit  Programme  [5,8].  The  aim  in  improving  the  ability  of  an  autonomous  vehicle  to  handle 
higher  level  functions  and  tasks  and  environments  with  significant  levels  of  uncertainty  associated  with  them, 
can  be  expressed  directly  in  terms  of  increasing  the  PACT  level  of  tasks  from  that  currently  attainable. 
As  an  example,  Figure  5.35  provides  a  possible  PACT  Contract  Level  diagram  for  a  set  of  ‘typical’  tasks, 
in  4  groups,  over  a  6  phase  mission  for  an  intermediate  type  of  autonomous  underwater  vehicle,  as  in 
Figure  5.35.  Those  tasks  with  a  PACT  level  below  3  do  not  involve  suggested  courses  of  action  from  the 
vehicle  for  the  human  controller;  they  require  direct  initialisation  by  that  controller.  The  PACT  Scheme 
provides  a  useful  tool  in  aiding  this  process,  allowing  a  direct  visualisation  of  the  dynamically  changing 
autonomy  structures  within  the  system. 
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Figure  5-35:  Possible  PACT  Contract  Levels  for  a  UUV  Mission. 


5.6.3.4  Multi-Agent  System  Representation 

Recent  work  addressing  dynamically  adaptive  autonomy  in  multi-agent  systems  [12,13],  puts  forward  a 
framework  that  permits  agents  to  dynamically  form,  modify  or  dissolve  goal-oriented  problem  solving  groups. 
In  particular,  it  enables  these  agents  to  do  so  in  the  presence  of  high  levels  of  change  and,  ultimately, 
uncertainty.  It  provides  them  with  the  key  ingredients  that  permit  the  transition  from  adaptability  to  adaptation 
-  motivation  and  the  ability  to  change  the  problem-solving  framework  of  the  system  itself. 

In  addition,  [12]  introduces  the  concept  of  the  sensible  agent,  which  participate  in  a  two  phase  process  prior  to 
engaging  in  tasks,  namely: 

1)  A  decision-making  phase,  during  which  sub-tasks  designed  to  carry  out  the  goal  are  identified  and 
agreed  upon;  and 

2)  A  task  allocation  phase,  during  which  the  various  agents  are  assigned  actions  and  tasks,  in  accordance 
with  the  decisions  made. 

The  point  is  made  that  agents  using  any  autonomy  model  must  comprehend  the  concepts  of  ‘self  and  ‘others’. 
An  example  would  be  the  sensor  system  processor  cooperating  with  the  navigation  system  processor  to 
prevent  the  vehicle  running  aground  or  colliding  with  an  obstacle.  To  achieve  this,  each  agent  must  possess  its 
own  environmental  model,  such  that  it  can  understand  the  implications  of  the  information  it  is  receiving  for 
any  of  the  other  agents  and  act  accordingly. 
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A  constrained  version  of  this  model  can  be  applied  to  the  case  of  multi -processor  platforms,  such  as  UUVs 
(e.g.,  Manta),  where  certain  actions  must,  by  definition,  be  performed  by  a  particular  agent  (i.e.,  processor) 
and  where  there  is  a  requirement  for  a  single  focus  for  enacting  the  overall  mission  plan. 

This  is  most  conveniently  achieved  via  the  fact  that  UUVs  normally  possess  a  central,  or  main,  processor, 
which  can  be  permitted  to  act  as  arbiter  in  task  assignment  and  decision-making  in  general,  fulfilling,  as  such, 
the  role  of  a  local  master. 

It  can  also,  where  operational  conditions  permit,  facilitate  the  dissemination  of  relevant  information  between 
agents,  to  augment  their  common  knowledge  base  (i.e.,  corporate  knowledge).  Moving  beyond  the  level  of  the 
individual  platform,  within  the  context  of  Network  Enabled  Capability  (NEC),  this  approach  may  easily  be 
scaled  so  as  to  encompass  cooperation  among  any  number  of  unmanned  platforms,  in  order  to  achieve  a 
common  mission  objective. 

5.6.3.5  Autonomous  Underwater  Decision  Making 

Dstl  work  has  provided  a  simulation  of  key  aspects  of  the  core  UUV  functionality  in  an  “unattended  cognitive 
underwater  vehicle”  (UCUV)  autonomy  project.  This  work  is  intended  to  provide  partial  proof  of  concept 
demonstration  of  specific  autonomous  decision  making  activities  by  unmanned  vehicles  using  a  navigational 
exercise.  This  work  is  in  its  early  stages  of  development  and  there  is  a  long  way  to  go  to  realise  the  ultimate 
aim  of  a  wholly  autonomous  cognitive  underwater  vehicle.  Figure  5.36  provides  an  overview  of  the  various 
inputs  and  contributing  components  for  this  modelling  process. 
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Figure  5-36:  Tools  and  Techniques  Drawn  upon  in  the  DSTL  UCUV  Work. 


The  work  on  UCUV  artificial  intelligence  engine  development  has  two  facets: 

1)  The  development  of  a  knowledge  based  system  for  “inside-the-envelope”  decision  making  [14];  and 

2)  The  development  of  a  Bayesian  based  learning  techniques  for  “outside-the-envelope”  decision 
making  [15]. 

In  order  to  capture  a  capability  to  handle  ‘within  the  envelope’  higher  level  decision  making  (i.e.,  based  on 
direct  knowledge),  a  core  knowledge  base  has  been  constructed  to  encompass  a  subset  of  the  relevant  drivers 
and  principles  involved  [14].  Initial  knowledge  acquisition  (KA)  focussed  on  UUV  navigating  to  a 
given  target  (identified  by  Rapidly  Deployable  Sonar)  whilst  avoiding  charted  objects  en  route. 
Using  CommonKADS  knowledge  modelling  methodology  [16,17],  further  KA  enabled  knowledge  models  to 
be  constructed,  from  which  an  intelligent  routing  application  was  specified,  designed  and  implemented. 

In  a  fully  autonomous  system,  without  recourse  to  guidance  from  a  human  operator,  all  of  the  problems  that 
may  be  encountered  must  be  resolved  on-board  and  an  appropriate  decision  made  in  all  cases.  So,  where  the 
situation  encountered  lies  ‘outside  the  envelope’  for  this  knowledge  base,  or  where  no  clear  cut  decision  can 
be  made  for  some  other  reason,  an  alternative  decision  engine  will  be  needed  to  apply  to  the  process,  to  force 
the  issue. 
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Based  upon  a  survey  of  the  relevant  literature,  a  Bayesian  learning  based  decision  engine  was  decided  upon 
for  addressing  these  “out  of  envelope”  situations  [15].  This  was  adopted  because  of  the  inherent  advantages  of 
the  approach  over  alternatives  such  as  neural  networks,  for  handling  ‘out  of  the  envelope’  situations,  where  a 
traditional  expert  system  type  approach  is  inapplicable.  Of  particular  utility  here  is  its’  robustness  in  the 
presence  of  high  levels  of  uncertainty.  An  incremental  variation  on  this  approach,  the  Bayesian  Agent,  was 
also  identified,  which  offers  a  capacity  for  both  on-line  and  off-line  learning  [18]. 

This  survey  also  identified  several  priors  of  potential  utility  here.  In  order  to  evaluate  these  various  options, 
a  simulation  exercise  was  conducted  on  narrow  channel  navigation  scenarios  for  a  simulated  estuary 
environment. 

The  work  employed  the  MATLAB  modelling  environment  since  this  permits  a  rigorous  top-down  design 
methodology  to  be  applied  to  the  construction  of  the  model,  for  configuration/life-cycle  management 
purposes.  This  system  also  permits  the  pull  through  into  C  code  of  the  model  components  directly  from  the 
model  itself.  Stateflow  is  a  part  of  the  MATLAB  suite,  which  allows  the  construction  of  block  based  models 
of  finite  state  machines,  through  the  Simulink  dynamic  simulation  system. 

A  stylised  simulated  estuarine  environment  (1000  m  x  500  m  x  25  m  deep),  populated  with  a  rough  terrain  bed 
and  traversed  by  a  simulated  narrow  navigation  channel,  cut  through  the  bed  to  the  bottom,  was  chosen  for 
reconstruction  in  simulation. 

As  a  result  of  an  extensive  literature  search,  two  alternative  Bayesian  based  approaches  were  selected  for 
comparison  (MML  and  BCPN).  What  does  this  mean?  Naive  Bayesian  learning  was  employed  on  the  network 
as  a  control  in  the  exercise.  In  addition,  an  initial  investigation  was  made  into  the  suitability  of  using  an 
ARTMAP,  What  does  this  mean?  neural  network  approach  to  selection  of  appropriate  prior  distributional 
forms  for  the  system  [19].  The  problem  that  was  posed  to  each  of  the  Bayesian  systems,  was:  Given  the 
current  direction  of  travel,  provide  the  system  with  any  changes  of  direction  that  are  required,  so  as  to 
maintain  a  course  that  is  as  close  to  the  centre  of  the  channel  as  possible,  at  all  times. 

To,  this  end  a  20  m  by  20  m  ‘window’  (i.e.,  the  width  of  the  channel)  was  used,  to  simulate  a  required 
clearway  for  safe  passage  of  the  craft  through  the  channel.  As  our  system  has  aim  grid  size,  with  each  of  the 
mesh  points  representing  the  centre  of  a  1  m  by  1  m  by  1  m  block,  this  translated  itself  into  a  19  mesh  point  by 
19  mesh  point  grid,  which  was  required  to  be  maintained  clear  of  solids.  For  this  exercise,  no  obstacles  were 
placed  upon  the  bed  of  the  channel.  However,  the  trained  classifier  can  be  used  with  equal  effectiveness  upon 
the  task  of  avoiding  obstacles  in  the  vertical  direction. 

All  three  of  the  Bayesian  learning  network  underpinned  approaches  investigated  can  provide,  within  certain 
limitations,  a  means  of  navigation  through  narrow,  sinuous  channels,  through  the  provision  of  suitable  course 
corrections.  The  introduction  of  signal  degradation  does  impact  on  the  performance  of  all  three  approaches, 
but  not  catastrophically  so  (graceful  degradation).  By  limiting  the  required  course  corrections  required  to  the 
range  of  +/-  3  (1  m)  grid  points  per  correction,  the  errors  in  the  corrections  supplied  all  lie  with  the  range  of 
+/-  1  grid  point  (rounded)  after  10  training  scenarios,  even  for  relatively  noisy  data  ([0.0, 0.4]  for  0.0,  [0.6, 1.0] 
for  1.0).  Due  to  the  much  higher  computational  requirements  for  the  MML  approach,  coupled  with  it  not 
providing  superior  performance  to  either  of  the  other  2  approaches,  it  is  not  considered  to  be  suitable  for  use 
for  such  a  task. 

The  use  of  an  ARTMAP  based  classifier  to  select  the  most  appropriate  prior  form  for  the  nodes  in  a  Bayesian 
network  shows  some  initial  promise.  Its  use  in  such  a  role  requires  further  investigation  though. 
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The  BCPNN  based  approach  [20,21]  was  recommended  for  pull  through  into  native  C  code  for  further  testing 
against  a  scenario  based  on  real  GIS  data.  This  was  trained  on  the  training  data  set,  with  applied  noise  levels 
giving  up  to  80%  reduction  in  signal  separation  between  land  and  water,  which  was  that  applied  to  the  signals 
for  this  test.  It  was  then  applied  to  a  compressed  section  of  the  Columbia  River,  providing  a  tortuous  path, 
with  frequent  narrowing  and  shallowing  to  below  the  20  m  diameter  of  the  training  data  channels.  This  test 
was  successful,  with  the  course  followed  being  within  1  m  of  the  mid  line  (i.e.,  1  grid  point). 

Overall,  the  test  of  the  BCPNN  approach  on  real  data,  based  on  a  section  of  the  Columbia  River, 
with  significant  ‘out  of  the  envelope’  behaviour,  in  terms  of  narrowing  and  shallowing,  was  a  success. 
This  shows  the  robustness  of  the  Bayesian  approach  itself  and  also  significant  promise  for  the  application  of 
the  ‘abstract  and  simplify’  approach  adopted  here,  in  the  context  of  noisy  (to  80%),  information-poor 
environments. 

The  task  was  extremely  challenging  since  this  was  a  new  domain  for  the  application  of  Bayesian  based 
decision  making  methods.  The  methodology  is  widely  applicable,  rather  than  domain  dependent. 

The  overall  aim  of  the  work  was  to  provide  an  initial  investigation  to  provide  proof  of  concept  of  decision 
making  in  support  of  autonomous  UUV  operation.  This  has  been  accomplished  successfully,  albeit  in  a 
limited  way.  The  project  needs  to  explore  and  illustrate  the  use  of  the  developed  methodology  on  a  wider 
subset  of  the  potential  problem  set. 

5.6.3.6  Conclusions 

The  demonstration  of  the  robust  applicability  of  the  principles  and  approach  adopted,  to  the  solution  of 
realistic  and  relevant  problems,  certainly  indicates  its  potential  worth.  This  work  is  innovative  in  its  very 
nature,  particularly  within  the  field  of  autonomous  decision  making  for  unmanned  vehicles.  It  forms  an 
exploration  of  the  possible.  However,  it  is  an  exploration  based  firmly  on  sound  mathematical  and  statistical 
principles.  There  is  a  need  to  investigate  the  application  of  the  approach  and  implemented  algorithms  to  other 
important  noisy  and/or  information  poor  decision  situations,  within  the  context  of  projected  UUV  operational 
scenarios.  Ultimately  there  is  a  need  to  address  higher-level  decision  making  tasks  in  order  to  provide  an 
integration  and  a  synthesis  of  the  processes,  especially  within  the  context  of  the  future  network  enabled 
environment. 
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Chapter  6  -  ADVANCED  UMV  OPERATOR  INTERFACES 

Chapter  Lead:  M.  Draper 

Contributors:  T.  Barry,  G.  Calhoun,  J.  Clark,  M.  Draper,  M.  Goodrich,  C.  Jansen, 

J.  Kessens,  F.  Kooi,  A.  Lefebvre,  S.  Murray,  J.  Nelson,  C.  Nielsen,  G.  Osga,  A.  Oudenhuijzen, 
M.  Quigley,  R.  Shively,  B.  Simpson,  R.  Stone,  L.  van  Breda,  J.  van  Delft,  J.  van  Erp 

6.1  INTRODUCTION 

Even  with  rapid  advances  in  computer  processing,  automation  technology,  and  artificial  intelligence  methods, 
there  remains  a  critical  need  for  human  involvement  in  order  for  UMVs  to  successfully  perform  their 
missions.  The  human  provides  unique  strategic  and  innovative  decision-making  capabilities  within  complex, 
dynamic,  and  time  sensitive  situations.  UMV  operator  performance  and,  by  extension,  the  UMV  operator 
control/display  interface,  will  be  even  more  critical  to  achieving  anticipated  new  and  increasingly  complex 
UMV  capabilities  including  close-coupled  operations  with  manned  systems,  UMV  interoperability, 
and  military  strike/combat  operations. 

Given  that  humans  are  to  remain  a  key  component  of  UMV  systems  for  the  foreseeable  future,  it  is  important 
to  recognize  the  unique  challenges  levied  upon  the  operator.  These  challenges  include  the  effects  of  system 
time  delays  (both  fixed  and  variable),  bandwidth  limitations  (which  can  be  intermittent),  datalink 
degradations/dropouts,  and  the  loss  of  the  rich  supply  of  multi-sensory  information  often  afforded  to  onboard 
operators.  With  future  highly  automated  UMV  systems,  issues  also  include  functional  allocation  of  tasks 
between  the  operator  and  the  system,  human  vigilance  decrements,  ‘clumsy  automation’,  limited  system 
flexibility,  mode  awareness,  trust/acceptance  issues,  failure  detection,  and  automation  biases.  Additional 
challenges  have  been  documented  in  detail  elsewhere  (Chapter  2).  However,  it  is  also  important  to  note  that 
the  physical  separation  of  crew  from  vehicle  might  also  offer  some  unique  benefits  that  should  be  exploited. 
Besides  the  obvious  benefit  to  crew  safety,  it  is  quite  likely  that  available  bandwidth  and  the  variety  of 
available  information  sources  might  be,  in  certain  cases,  far  greater  for  a  geographically-separated  UMV  crew 
versus  an  onboard  operator,  potentially  resulting  in  more  situation  awareness  rather  then  less.  This,  of  course, 
assumes  that  a  well-designed  operator  interface  exists  that  can  rapidly  filter  and  fuse  this  expanded 
information  into  intuitive  displays,  again  underscoring  the  need  to  attend  to  operator  interface  issues  to  ensure 
maximal  system  performance. 

It  is  also  important  to  note  that  as  technology  advances,  the  role  of  the  UMV  operator  must  change  as  well. 
Therefore,  UMV  operator  interfaces  should  not  be  considered  ‘one-size-fits-alT,  but  must  be  tailored  to  match 
the  capabilities  and  limitations  of  the  host  system  and  intended  mission.  Most  current  UMVs  require  that 
operators  have  the  capability  to  manually  control  the  vehicle  and  activate  state  changes  (i.e.,  direct  tele¬ 
operation).  Thus,  operator  interfaces  for  these  vehicles  can  best  leverage  the  numerous  lessons  learned  from 
decades  of  inner-loop  control  design  research,  while  applying  novel  interfaces  to  combat  challenges  that  are 
uniquely  associated  with  UMV  operation. 

With  new,  highly  automated  UMVs,  the  operator’s  role  is  becoming  more  supervisory  in  nature,  overseeing 
the  automated  activation  of  programmed  events  (e.g.,  making  sure  the  appropriate  event  is  activated  at  the 
appropriate  time),  managing  changes  to  the  automated  mission  plan,  and  making  more  strategic-level 
decisions.  These  operator  interfaces  must  take  into  account  issues  associated  with  automation  management, 
including  vigilance  effects,  brittle/clumsy  automation,  sudden  workload  spikes,  etc. 
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Continuing  this  trend  beyond  the  current  state-of-the-art,  a  vision  exists  for  a  new  interface  paradigm  for 
controlling  next  generation  UMVs.  This  envisioned  interface  system  involves  multiple  semi-autonomous 
UMVs  being  controlled  by  a  single  supervisor.  These  UMVs  will  have  the  capability  to  make  certain  higher- 
order  decisions,  independent  of  operator  input  and  pre-defined  mission  plans.  This  capability  of  the  UMV 
‘to  decide’  constitutes  a  whole  new  set  of  challenges  for  operators,  as  they  will  be  required  to  rapidly  judge 
the  appropriateness  of  these  decisions  and  assess  their  impact  on  overall  mission  objectives,  priorities, 
etc.  Future  operator  interfaces  will  need  controls  and  displays  tailored  for  multi-UMV  control  and  must  allow 
the  operator  the  capability  to  easily  inspect/override  the  autonomous  UMV  decision-making  logic. 
These  interfaces  will  also  need  to  provide  information  fusing/filtering  algorithms,  intelligent 
prioritization/cueing  logic,  and  possibly  some  form  of  adaptive  task  allocation  in  response  to  rapidly  changing 
events  and/or  workload  levels. 

This  chapter  explores  many  relevant  issues  surrounding  UMV  control/display  interface  technology. 
The  chapter  begins  with  a  discussion  of  the  importance  of  pursuing  a  user-centered  design  methodology  in 
designing  UMV  operator  interfaces.  This  is  followed  by  a  section  detailing  available  and  applicable  design 
guidelines  as  well  as  noticeable  gaps  in  this  area.  Next,  several  candidate  data  input  and  display  technologies 
are  discussed  in  turn.  Each  technology  section  (with  few  exceptions)  begins  with  a  summary  description  of  the 
technology  area  along  with  any  commonly  accepted  subcategories  of  that  technology.  Next,  there  is  a 
discussion  of  actual  or  potential  application  of  the  technology  to  UMVs.  Lastly,  technology  maturity, 
challenges,  and  unresolved  issues  are  summarized.  The  chapter  then  focuses  on  operator  interface  issues 
associated  with  multi-UMV  supervisory  control  and  concludes  with  additional  UMV  interface  considerations 
such  as  scale,  mobility,  and  various  communication  constraints. 


6.2  USER-CENTERED  DESIGN  FOR  UMV  CONTROL 

UMVs  operating  in  collaborative,  semi-autonomous  groups  represent  a  major  advance  in  military  capability. 
Multiple  remote  platforms  conducting  operations  in  hazardous  environments  -  where  errors  can  have  serious 
military  and  political  consequences  -  also  represent  a  new  level  of  engineering  complexity.  The  missions 
planned  for  UMVs  are  diverse  and  complex  by  any  standard;  the  control  logic  required  to  operate  them  will 
also  be  complex  and  will  always  contain  an  element  of  risk. 

Because  UMVs,  like  all  military  systems,  are  extensions  of  human  intent,  UMV  effectiveness  can  only  be  as 
good  as  the  clarity  of  that  intent.  That  intent  is  conveyed  via  the  primary  tool  for  remote  human  control  - 
the  user  interface.  Good  control  depends  on  good  information  -  which  must  be  complete,  sensible,  reliable, 
and  timely.  The  quality  of  the  user  interface,  therefore,  will  ultimately  determine  the  quality  of  system 
performance.  A  good  interface  can  help  the  human  controller  to  maintain  situation  awareness  of  the  remote 
environment  where  UMVs  are  operating,  can  enhance  goal  setting  under  dynamic  vehicle  and  mission 
conditions,  and  can  optimize  human  analytical  processes  and  decision-making. 

Certainly,  a  lot  is  known  about  how  to  provide  the  human  controller  with  good  information  and  decision 
support  tools  -  the  challenge  is  to  apply  this  knowledge  aggressively  and  early  in  the  design  process.  Human 
factors  engineering  (HFE)  methods  inform  the  interface  design,  i.e.,  its  content,  layout,  and  interaction 
metaphor,  while  human-system  integration  (HSI)  methods  place  the  interface  in  the  context  of  the  total 
human-UMV  system  including  its  missions,  operating  environment,  and  support  requirements. 

Human  factors  engineering  defines  the  form  and  behavior  of  the  controller  interface.  It  applies  knowledge  of 
human  perceptual  capabilities,  motor  skills,  memory  capacity,  decision-making  processes  and  communication 
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styles  to  the  design  of  interface  functions.  Complex  applications  involving  teams  of  operators  must  also 
address  human  characteristics  related  to  shared  human  goals,  team  interactions,  and  distributed  decision¬ 
making.  The  primary  objectives  of  HFE  is  to  ensure  that  information  presented  to  a  controller  matches  their 
cognitive  and  decision-making  requirements  in  real  time,  that  the  information  is  easily  understood,  and  that 
controller  decisions  are  easily  implemented. 

Human-system  integration  is  a  critical  component  of  the  systems  engineering  discipline  that  addresses  human 
capabilities  and  limitations  throughout  the  UMV  development  process.  The  objective  of  HSI  is  to  match 
technology  to  user  capabilities  in  order  to  optimize  mission  performance.  HSI  also  employs  HFE  methods  and 
data,  but  applies  them  to  broader  issues  of  safety,  training,  and  to  technology  or  mission  evolution  over  the 
system  life  cycle.  “Integration”  implies  a  task  conducted  amid  multiple,  often-conflicting  system  design 
demands  so  HSI  must  address  such  issues  as  the  impact  of  network  communications  architectures,  varying 
levels  of  UMV  automation  capability  (including  intra- vehicle  communications  and  decision-making),  and  the 
human  role  in  coping  with  degraded  conditions.  HSI  methods  can  ensure  that  systems  are  built  to 
accommodate  the  characteristics  of  the  personnel  who  will  operate,  maintain,  and  support  them. 

HFE  and  HSI  tools  and  methods  are  highly  complementary  and,  insofar  as  they  reflect  advocacy  of  the  human 
role  in  system  design,  may  be  combined  into  an  overarching  concept  of  user-centered  design  (UCD). 
The  UCD  approach  reflects  an  appreciation  of  human  potential  in  the  system  engineering  process.  Regardless 
of  the  engineering  sophistication  of  any  military  system,  human  controllers  -  with  their  unique  abilities  to 
sense  and  understand  both  the  system  and  the  mission  -  have  always  added  the  critical  margin  of  performance 
that  makes  such  systems  viable.  The  UCD  orientation  to  UMV  development,  therefore,  represents  a  critical 
contribution  to  the  success  of  such  complex  systems. 

The  new  capabilities  reflected  by  UMV  systems  will  stress  current  engineering  design  practice  regarding 
control  logic,  communications  networks,  and  multi-level  automation.  These  same  capabilities  will  stress 
current  perspectives  of  how  human  presence  can  best  support  the  new  missions  and  expanded  mission  modes 
enabled  by  UMVs.  UCD,  therefore,  will  be  faced  with  new  challenges  that  will  extend  our  concepts  of 
human-system  interaction  to  settings  where  machines  are  cast  (at  a  behavioral  level)  more  as  colleagues/ 
collaborators  than  as  mere  tools  in  military  operations: 

•  New  models  of  human-machine  and  human-human  communication  must  be  elaborated,  especially  in 
the  context  of  network-centric  warfare  doctrine. 

•  New  approaches  to  smoothly  and  safely  altering  multi-agent  control  configurations  (the  consequence 
of  enhanced  automation  capabilities)  must  be  defined. 

•  Because  UMV  control  should  be  achievable  anywhere  -  from  the  fixed-base  command  center  to  the 
mobile,  individual  warfighter  -  scaleable  interface  designs,  based  on  formal  HFE/HSI  principles, 
must  be  generated. 

•  As  hardware  and  mission  concepts  change  over  time,  UCD  methods  must  accommodate  the  certain 
requirement  for  graceful  (rather  than  disruptive)  life  cycle  evolution. 

•  Finally,  new  testing  and  validation  methods  must  be  generated  and  proven  to  ensure  that  the  resulting 
systems  will  perform  as  required. 

UCD  has  a  clear  and  historically  proven  role  in  good  system  design.  Furthermore,  when  explicitly  included  at 
the  beginning  of  a  sound  systems  engineering  program,  UCD  can  be  accommodated  without  interference  to 
other  engineering  tasks.  While  UMV  system  design  must  address  new  challenges  across  a  broad  front  of 
engineering  specialties,  UCD  integration  is  an  enhancing  -  rather  than  distracting  -  step  in  such  an  ambitious 
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undertaking.  An  irony  of  many  military  acquisition  programs  is  that  UCD  is  never  deleted;  some 
consideration  to  HFE  and  HSI  issues  is  almost  invariably  required  either  at  the  time  of  system  fielding  or 
shortly  thereafter,  when  performance  problems  force  an  examination  of  the  human  role  in  mission 
effectiveness  -  usually  with  increased  cost  or  delay.  UMV  design  can  expand  both  our  engineering  and  human 
interface  design  knowledge,  and  can  add  to  the  technical  “toolboxes”  of  both  domains.  To  realize  the  full 
potential  of  the  UMV  concept,  however,  UCD  design  methods  must  be  fundamental  to  the  engineering 
process  from  its  initiation. 


6.3  GUIDELINES  AND  GAPS 

Since  the  advent  of  highly  capable  uninhabited  vehicles,  notably  in  the  application  domains  of  defence, 
offshore  oil  and  gas  exploration,  attention  has  increasingly  focused  on  the  development  of  technologies 
necessary  to  endow  vehicles  with  complete  autonomy.  This  approach  has  not  met  with  widespread  success. 
Evidence  frequently  points  to  the  fact  that  the  human  operator  still  has  a  significant  role  to  play  in  the  future 
of  uninhabited  vehicles,  as  part  of  a  control  continuum  that  ranges  from  direct  tele-operation  during  critical 
mission  phases  and  recovery  modes  to  the  high-level  supervision  of  single  or  multiple  platforms.  However, 
few  (if  any)  usable  guidelines  and/or  experimental  test  beds  exist  to  help  ensure  that  human  factors  issues  are 
taken  into  account  early  in  the  design  lifecycle  of  uninhabited  vehicles  and  their  control-display  interfaces 
with  the  human  operator  or  supervisor. 

6.3.1  Pre-Requisites  to  Selecting  Human  Interface  Devices 

The  market  for  off-the-shelf  interface  devices  worthy  of  consideration  for  UMV  application  is  rapidly 
growing.  In  particular,  the  exploitation  of  games  console  hand  controller  designs  is  gaining  pace,  as  there  is  a 
perception  (unproven  at  the  time  of  writing)  that  new  recruits  to  the  Armed  Forces  will  adapt  “seamlessly”  to 
advanced  military  interfaces  if  those  interfaces  are  based  on  familiar  gaming  devices.  Despite  this,  an  early 
human-centred  design  approach  to  device  specification/design/procurement  is  essential;  any  display  or  control 
device  must  be  selected  on  the  basis  of  a  thorough  review  of  the  tasks  expected  of  the  human  operator, 
regardless  of  whether  his/her  role  is  that  of  flight/mission/payload  specialist,  supervisor  or  field  dispatch 
(see  Section  6.2). 

The  pitfalls  of  being  swayed  by  technology  push  are  now  well  documented  throughout  the  international 
human  factors  community.  Obsolete  and  unused  Virtual  Environment  Centres  bear  witness  to  the  hazards  of 
selecting  technologies  before  the  needs  of  the  end  user  population  have  been  fully  understood.  Yet  still  one 
finds  examples  where,  whilst  the  information  to  be  displayed  to  the  human  user  has  been  subjected  to  strong 
human  factors  principles,  those  same  principles  are  ignored  in  what  is  blatantly  a  prescriptive  selection  of 
unproven  human  interface  technologies.  This  situation  is  particularly  rife  in  the  virtual  reality/multi-sensory 
interface  arena.  Here,  advanced  display  techniques  are  often  adopted  only  as  a  result  of  their  “high-tech” 
qualities.  However,  when  implemented  in  a  tele-operation  or  telepresence  system,  those  same  qualities  could 
then  limit  the  utility  of  the  system,  because  the  human  user  spends  more  time  trying  to  adapt  to  the  limitations 
of  the  interface  technology,  rather  than  benefiting  from  the  displayed  content  of  the  application  itself. 

To  overcome  this  problem,  the  interactive  facilities  for  UMV  supervision  and  control  must  evolve  from  the 
early  performance  of  a  hierarchical  task  analysis  (at  least),  and  a  domain-specific  analysis  (at  best;  e.g.,  [1]). 
The  interactive  facilities  must  also  take  into  consideration  the  domain-specific  issues  covered  by  the  task 
analysis  (and  the  subject  of  Static/Dynamic  Allocation  of  Function  specifically  (e.g.,  [2, 3, 4, 5])  and  must  take 
into  consideration  the  human  factors  limitations  of  candidate  interface  technologies  (e.g.,  [1]). 
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6.3.2  Standards/Methodologies/Best  Practice  and  Gaps 

Academic  and  military  laboratory-based  RandD  programmes  have  often  failed  to  generate  realistic  interface 
design  guidelines  to  support  the  human  operator  in  controlling  UMVs  safely  and  efficiently.  Furthermore, 
the  international  standards  community  also  admits  that,  as  far  as  advanced  computer-based  human  interfaces 
are  concerned,  it  is  further  behind  in  the  delivery  of  underpinning  standards  for  new  technological 
developments,  such  as  unmanned  systems,  than  it  would  like  to  be. 

There  have  been  a  few  attempts  historically  to  collate  information  of  relevance  to  the  hazardous  environment 
tele-operation  community.  As  well  as  a  highly  relevant  Oak  Ridge  publication  [6],  UK  research  in  the  early 
1980s  led  to  the  production  of  a  Human  Factors  Design  Handbook  [7].  This  handbook  was  specifically 
concerned  with  the  design  of  human  interfaces  for  Remotely  Operated  (submersible)  Vehicles  (ROVs)  for  the 
North  Sea  oil  and  gas  industry  and  was  used  to  design  consoles  for  some  vehicles  still  in  service  today. 
Whilst,  today,  this  document  is  somewhat  out  of  date  (especially  with  regard  to  informational  displays  and  the 
combination  of  video  images  with  alphanumeric  data  or  forms  of  symbology  to  assist  in  navigation),  it  still 
contains  some  information  of  relevance  to  general  ergonomics  practice,  workplace  and  hand  control  design. 

Another  report  that  was  produced  with  remote  operations  in  mind  focused  on  the  needs  of  the  nuclear  industry 
and  developments  in  support  of  mobile  and  manipulator  systems  for  irradiated  facilities  [8].  Again,  some  of 
the  information  is  dated.  However,  sections  on  hand  control  design  and  the  application  of  recent  (1990s) 
interactive  technologies  (e.g.,  from  the  virtual  environments  community)  are  still  relevant. 

More  recently,  organisations  have  been  active  in  promoting  the  need  to  collate  human  factors  information  of 
relevance  to  the  unmanned  or  remotely  operated  vehicle  arena.  A  publication  [9]  was  written  to  fulfil  the  role 
of  an  informative  annex  to  pre-contractual  and  contractual  documents  relating  to  the  design  of  future  military 
or  explosive  ordnance  disposal  (EOD)  tele-operated/robot  systems. 

For  the  foreseeable  future,  then,  tele-operated  vehicle  and  manipulator  system  designers  will  have  to  rely  on 
general  ergonomics  and  human  factors  texts,  as  adopted  by  the  civilian  and  military  human  factors 
community.  This  is  far  from  satisfactory  because  such  texts  tend  to  be  outdated  before  they  are  even 
published.  A  good  example  here  is  the  current  generation  of  ISO  standards  pertaining  to  human-computer 
interaction,  such  as  ISO  14915,  which  does  not  adequately  address  the  needs  of  the  growing  multi-modal  and 
synthetic  environments  communities.  Furthermore,  the  refresh  cycles  for  these  documents  are  very  long-term 
indeed.  Some  standards,  such  as  DEF  STAN  00-25  in  the  UK  {Human  Factors  for  Designers  of  Equipment ) 
are  only  now  being  considered  for  updating  (or  “future  proofing”)  -  many  of  the  contents  dating  back  to,  for 
example,  [10]. 

Ageing  and  irrelevant  contents  of  existing  texts  also  force  the  authors  of  contemporary  standards  and 
guidelines  to  take  short  cuts  when  attempting  to  make  their  documents  relevant  to  a  particular  domain  or 
system.  One  notable  example  of  this  is  Appendix  B3  (Human  Computer  Interface)  of  STANAG  4586  - 
Standard  Interface  of  the  Unmanned  Control  System  (UCS)  for  NATO  UAV  Interoperability.  Here,  the 
absence  of  any  appropriately  packaged  knowledge  relevant  to  UMV  human  interface  design  (and  relevant 
knowledge  certainly  exists  in  the  UMV  community),  has  forced  the  authors  to  use  unmodified  extracts  from 
ISO  9241  -  Ergonomic  Requirements  for  Office  Work  with  Visual  Display  Terminals. 

At  a  very  high  level,  then,  the  human-system  interfaces  provided  as  part  of  an  integrated  UMV  system, 
regardless  of  the  environment  on  which  that  vehicle  is  to  be  deployed  (land,  air,  sea  surface,  subsea,  space, 
etc.)  must,  at  their  most  basic  level,  be  capable  of: 
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•  Presenting  the  operator  with  appropriate  (video  and/or  synthetic)  imagery  from  the  remote 
environment  to  enable  situation  awareness,  waypoint  planning,  real-time  control  (driving,  flying, 
etc.),  obstacle  avoidance  (including  friendly  forces  or  civilians),  payload  selection,  payload  operation 
and  general  manipulation  to  occur  safely  and  efficiently; 

•  Presenting  the  operator  with  appropriate  real-time  data  to  enable  him/her  to  transition  seamlessly 
between  supervisory  and  tele-operation  modes  of  control  without  significant  loss  of  situation 
awareness  (encompassing  the  status  of  both  the  vehicle  -  power,  performance,  command  link 
integrity,  “health”  and  other  subsystems  -  and  the  remote  environment); 

•  Providing  the  operator  with  control  and  data  input  devices  appropriate  for  remote  control, 
manipulation,  subsystem  and  payload  selection/operation  tasks;  and 

•  Presenting  the  human  operator  with  mission-specific  data,  adopting  formats  that  avoid  any  cognitive 
conflict  or  mental  resource  monopolisation  ,  thereby  compromising  real-time  control  or  supervision. 

These  presentational  requirements  must  not  be  hindered  in  any  way  by  the  selection  and/or  design  of  the 
human  interface  hardware,  and  displays  and  controls  must  support  the  human  operator’s  intuitive  exploitation 
of  the  information  he/she  is  provided.  More  detailed  design  principles  and  guidelines,  including  reference  to 
the  development  of  UMV  workstations  and  portable  consoles  (originally  written  in  support  of  human  interface 
design  for  unmanned  EOD  vehicles)  can  be  found  in  [9]. 

Where  appropriate,  and  in  the  absence  of  access  to  physical  UMV  assets  or  appropriate  deployment  scenarios, 
Virtual  or  Synthetic  Environment  (VE/SE)  test  beds  should  be  given  serious  consideration  in  the  quest  to  fill 
the  human  factors  knowledge  gaps  in  both  interface  technology  provision  and  in  the  definition  of  the  role  of 
the  human  operator  in  supervising  or  directly  controlling  one  or  more  UMV  systems.  Early  examples  of  such 
test  beds,  based  on  low-cost  approaches  to  simulation  (e.g.,  using  Microsoft  DirectX,  .NET  or  games  engine 
technologies)  are  available.  The  Alchemy  experimentation  system,  for  example,  developed  as  part  of  the 
UK  Human  Factors  Integration  Defence  Technology  Centre  programme,  supports  Human  Factors 
investigations  of  advanced  display  and  control  devices  for  UMV  interaction,  from  head-mounted  displays  to 
video  console-like  hand  controllers  [11].  Alchemy  2  (Figure  6-1),  the  latest  version  of  this  test  bed, 
has  evolved  from  a  programming-intensive  environment  to  become  a  highly  reconfigurable  “serious  gaming” 
implementation  and  has  already  been  demonstrated  in  the  context  of  an  iSTAR  UAV  (Allied  Aerospace) 
and  “marsupial”  (SPAWAR)  land  vehicle  deployment  (tele-operated),  together  with  a  new  man-portable, 
twin  turbofan  UAV  concept  (Kestrel  Aerospace,  UK). 


Figure  6-1:  Alchemy  2  Experimentation  System. 
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Machine  Vision  (Contract  No.  C4004-20747). 

[10]  Van  Cott,  H.P.  and  Kinkade,  R.G.  (Eds.,  1972).  Human  Engineering  Guide  to  Equipment  Design. 
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[11]  Stone,  R.J.  (2005).  “Project  Alchemy:  An  Experimental  Human  Factors  Test  Bed  for  iSTAR  UAVs  in 
Urban  Operations”.  Proceedings  of  HCII  2005;  Las  Vegas. 

6.4  DATA  INPUT  TECHNOLOGIES 

6.4.1  Manual  Input  Devices  (keyboards,  mice,  control  sticks,  touch  devices,  etc.) 

6.4.1. 1  Description  of  Technology 

Controllers  that  involve  a  manual  input  include  single-purpose  buttons,  levers,  and  joysticks  as  well  as  a 
keyboard  for  text  entry  and  a  mouse  for  pointing  and  selecting  text  and  icons  presented  on  a  display. 
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Trackballs  and  touch  pads  can  also  perform  much  the  same  function  as  a  mouse.  Since  these  control  devices 
are  commonly  used  in  a  variety  of  applications,  no  further  description  will  be  given  herein.  A  less  frequently 
employed  manual  control  device  employs  touch  sensitive  technology  in  which  operators  press  the  display 
surface  over  the  appropriate  label  or  icon  presented  on  a  display  to  make  a  selection.  A  directory  of 
manufacturers  of  touch  screen  devices  is  available  [1]. 

Touch  sensitive  displays  can  sense  that  the  display  is  being  touched  by  a  pen  or  finger  and  can  send  this 
information,  along  with  the  touch  location,  to  a  host  controlling  system.  A  variety  of  technologies  have  been 
employed  to  implement  touch  screens:  resistance,  optical,  capacitive,  and  acoustic  techniques  [2].  These 
technologies  can  be  grouped  into  two  classes:  touch  screens  that  use  an  overlay  that  responds  to  pressure  and 
screens  that  are  activated  when  the  finger  or  pointer  interrupts  a  signal.  All  implementations  are  considered 
robust  and  operation  is  simple  and  easy  to  learn.  As  a  more  “direct”  selection  input  method,  a  mechanical 
intermediary  (mouse)  is  not  needed;  nor  is  there  need  for  positional  feedback  (cursor)  consuming  display 
space.  Rather,  the  touch  screen  serves  as  both  an  input  and  output  device,  capitalizing  on  direct  eye-hand 
coordination. 

6.4.1.2  Actual  or  Potential  Applications  to  UMVs 

In  existing  UMV  control  stations,  operators  typically  employ  the  manual  control  devices  just  described, 
with  the  exception  of  the  touch  screens.  There  is,  however,  potential  UMV  applications  of  touch  screens, 
especially  in  future  supervisory  control  stations  that  feature  more  automated  control  of  vehicle  and  camera 
movement.  Since  the  operator  will  have  less  dedicated  hands-on  (e.g.,  stick  and  throttle)  control  in  these 
stations,  the  majority  of  operator  interaction  will  involve  monitor  and  control  of  subsystems.  Thus,  touch 
screens  have  a  potential  application  to  these  UMV  stations,  as  function  selection  using  touch  screens  has 
proven  to  be  fast  and  accurate,  as  well  as  easy  to  learn  in  other  ground-based  applications  [3].  Moreover, 
since  the  number,  shape,  size,  location,  and  label  of  the  touch-sensitive  fields  are  under  software  control 
(compared  to  electromechanical  controls),  they  can  be  easily  reconfigured  corresponding  to  periodic  upgrades 
made  in  the  UMV  system  control  architecture.  It  should  be  noted,  though,  that  touch  screens  are  only 
appropriate  when  inputs  are  limited  and  well  defined  [4].  Touch  screens  are  not  suited  for  tasks  requiring 
precise  positioning  such  as  drawing  and  graphical  input.  Additional  guidelines  on  the  design  of  touch  screens 
(touch-selection  strategies,  button  size,  feedback  strategies,  mouse-emulation  strategies,  touch  biases  screen 
angles  [5]  and  numeric  keypads  [6])  are  available  for  use  in  applying  touch  screens  for  UMV  stations. 

6.4.1.3  Technology  Maturity,  Challenges,  and  Unresolved  Issues 

In  that  manual  control  devices  have  historically  been  used  in  UMV  ground  control  stations,  technology 
maturity  is  not  an  issue.  Even  touch  screens  are  commercially  available  and  widely  used  in  banks,  restaurants, 
airports,  and  ground-based  command/control  stations.  Thus,  a  Technology  Readiness  Level  (TRL)  Rating  #6 
can  be  assigned  to  touch  screens,  since  this  input  device  has  been  demonstrated  in  relevant  environments. 
(Conventional  manual  devices  are  at  a  TRL  Rating  #9.)  Even  though  touch  screens  and  other  manual  devices 
are  appropriate  for  UMV  station  applications,  there  still  remain  challenges  in  determining  how  best  to  employ 
these  devices  in  future  supervisory  stations  such  that  the  controls  support  decision  making  or  execution  in  a 
timely  and  error-free  manner. 

Dedicated  (single-function)  manual  control  devices  need  to  be  easily  located,  grasped,  and  manipulated. 
All  the  information  and  control  devices  needed  for  a  particular  set  of  activities  should  be  in  close  proximity 
and  ideally  available  with  less  than  two  key  presses.  Proper  and  consistent  formats,  abbreviations,  symbol 
meaning,  control  assignments,  procedures  and  rapid  (<  0.2  seconds)  feedback  need  to  be  employed  so  the 
action  required  and  status  of  control  operation  is  intuitive  to  the  operator  [7]. 
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Human-engineered  design  of  multi-function  controls  is  also  needed  [8].  With  multi-function  controls, 
operators  select  menu  options  presented  on  a  control  station  display  using  buttons  on  the  periphery  of  the 
display  or  a  touch  sensitive  screen.  While  a  large  number  of  systems  can  be  controlled  via  one  control  device, 
there  is  a  danger  in  the  operator  spending  excessive  time  with  the  head  down  in  order  to  progress  through 
multiple  interface  layers  or  menus.  This  can  lead  to  distraction  from  the  primary  tasks  of  maintaining  situation 
awareness  and  supervising  UMV  control.  Besides  effective  labelling  so  the  function  associated  with  each 
switch  is  obvious,  navigation  between  the  menus  and  the  interactive  sequences  need  to  be  efficient. 

Function  selection  using  touch  screens  presents  additional  challenges.  Arm  fatigue  and  screen  obstruction  by 
the  hand  may  occur.  This  is  especially  true  with  pen  input,  along  with  the  hassle  of  having  to  pick  up  the 
fragile  pen  for  each  input.  Operators  must  be  more  attentive  to  visual  and  audio  feedback  due  to  the  lack  of 
kinesthetic  feedback  associated  with  manipulating  real  switches  and  knobs  [9].  Given  the  size  of  an  operator’s 
flnger/pointer  and  the  parallax  potential  of  the  screen,  selection  of  small  targets  or  closely  spaced  functions  is 
also  difficult.  There  are  also  limits  on  the  types  of  interactions  that  can  be  accomplished.  For  instance, 
selecting  and  dragging  objects,  inputting  precise  positioning,  rubber  band  line  drawing,  pop  up  menu 
selections  and  other  interactions  for  which  the  mouse  is  well  adapted  are  not  suited  for  touch  screen  operation. 
To  enable  these  more  complex  interactions,  the  touch  screen  needs  to  sense  the  degree,  or  pressure,  of  contact 
or  sense  multiple  points  of  contact,  besides  just  sensing  the  push  and  release  of  touch.  If  the  system  isn’t  able 
to  sense  at  least  two  levels  of  pressure,  then  auxiliary  devices  must  be  used  for  signalling  [9].  One  potential 
solution  is  to  use  different  sensor  densities  which  allow  the  creation  of  display  areas  with  different  resolutions 
[10]. 


6.4.1.4  References  for  Manual  Input  Devices 

[1]  Buxton,  B.  (2005).  A  directory  of  sources  for  input  technologies.  Available  at:  http://www.billbuxton. 
com/InputSources  .html#anchor693  93  6 

[2]  Bullinger,  H.-J.,  Kern,  P.  and  Braun,  M.  (1997).  Controls.  In:  Salvendy  (Ed),  Handbook  of  Human 
Factors  and  Ergonomics,  2nd  Edition,  New  York:  John  Wiley  and  Sons,  Chapter  21,  pp.  729-771. 

[3]  Sears,  A.  and  Shneiderman,  B.  (1991).  High  precision  touch  screens:  Design  strategies  and  comparisons 
with  a  mouse,  International  Journal  of  Man-Machine  Studies,  34,  593-613. 

[4]  Plaisant,  C.  and  Sears,  A.  (1993).  Touchscreen  interfaces  for  alphanumeric  data  entry.  In:  B.  Shneiderman, 
Ed.,  Sparks  of  Innovation  in  Human-Computer-Interaction.  Norwood,  NJ:  Ablex. 

[5]  Scott,  B.M.  and  Conzola,  V.  (1997).  Designing  touch  screen  numeric  keypads:  effects  of  finger  size, 
key  size,  and  key  spacing.  Proceedings  of  the  Human  Factors  and  Ergonomics  Society,  360-364. 

[6]  Lewis,  J.R.  (1993).  Literature  review  of  touch-screen  research  from  1980  to  1992.  IBM  Technical 
Report  54.694.  New  York:  IBM  Corporation.  Available  at:  http://drjim.Ocatch.com/touchtr.pdf#search= 
‘Beringer%20AND%20touch%20and%20input 

[7]  Lind,  J.H.  and  Burge,  C.G.  (1992).  Human  Factors  Problems  for  Aircrew-aircraft  Interfaces:  Where 
should  we  focus  our  efforts?  In  “Advanced  Aircraft  Interfaces:  The  Machine  Side  of  the  Man-Machine 
Interface”,  AGARD-CP-521,  3-1-3-12. 

[8]  Calhoun,  G.L.  and  Herron,  E.L.  (1982).  Pilot-machine  Interface  Considerations  for  Advanced  Aircraft 
Avionics  Systems,  In:  “Advanced  Avionics  and  the  Military  Aircraft  Man/Machine  Interface”,  AGARD- 
CP-329,  Chapter  24,  1-7. 
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[9]  Buxton,  W.,  Hill,  R.  and  Rowley,  P.  (1987).  Issues  and  Techniques  in  Touch-sensitive  Tablet  Input, 
Computer  Graphic,  19(3),  215-224. 

[10]  Westerman,  W.,  Elias,  J.G.  and  Hedge,  A.  (2001).  Multi-touch:  A  new  tactile  2-D  gesture  interface  for 
human  computer  interaction.  Proceedings  of  the  Human  Factors  and  Ergonomics  Society,  632-636. 

6.4.2  Speech  Recognition  Systems 
6.4.2. 1  Description  of  Technology 

Automatic  Speech  Recognition  (ASR)  technology,  which  transforms  an  operator’s  spoken  words  into  machine 
text,  has  been  around  for  over  50  years  [1].  Investigating  this  technology  for  use  in  military  systems  has  been 
ongoing  for  over  30  years.  In  the  early  1970s,  researchers  quickly  realized  that  along  with  the  rapidly 
advancing  computer  technology  came  the  equally  rapid  proliferation  of  knobs,  dials,  switches  and  other 
traditional  interface  devices  needed  by  operators  to  control  that  technology.  Not  only  would  these 
conventional  controls  begin  to  take  up  large  amounts  of  physical  space,  but  their  sheer  numbers  would  also 
quickly  overwhelm  the  human  operator’s  ability  to  effectively  manage  them  all,  especially  in  time  critical 
situations.  For  these  reasons,  researchers  began  to  explore  the  possibilities  of  using  ASR,  also  known  as  Direct 
Voice  Input  (DVI),  in  addition  to  speech  synthesis  technology,  which  translates  text  into  machine-spoken 
words,  to  control  and  display  information  [2,3,4]. 

Much  of  the  early  research  in  the  military  use  of  speech  recognition  technology  was  applied  to  the  single-seat 
aircraft  cockpit  [5, 6, 7, 8, 9]  where  pilots  must  not  only  use  stick,  throttle  and  rudders  to  keep  the  aircraft  in  the 
air,  but  must  also  manage  an  increasing  number  of  sophisticated  onboard  computer  systems  for  offensive, 
defensive,  surveillance,  and  a  myriad  of  other  tasks.  It  was  becoming  increasingly  clear  that  without  another 
crewmember  in  the  cockpit,  the  complexity  of  these  systems  would  eventually  overwhelm  the  single  pilot’s 
ability  to  effectively  manage  the  system.  If  an  accurate  DVI  system  were  used,  the  pilot  could  verbally  ask  the 
aircraft  to  accomplish  simple  tasks  like  changing  a  radio  frequency  or  navigate  to  a  new  waypoint  [10,11]; 
much  the  same  way  a  pilot  might  ask  his/her  co-pilot  to  accomplish  these  tasks. 

Since  1987,  the  United  States  Defense  Advanced  Research  Projects  Agency  (DARPA)  and  the  National 
Institute  of  Standards  and  Technology  (NIST),  have  sponsored  objective  benchmark  tests  in  the  DVI  research 
community  [12].  Figure  6-2  shows  how  the  state  of  the  art  of  the  technology  has  improved  significantly  in 
recent  years,  and  as  a  result,  is  rapidly  becoming  widely  accepted  by  users  [13]. 
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Figure  6-2:  NIST  Speech  Recognition  Benchmark  Test  History. 

Speaker  dependent,  discrete  word  recognition  systems  requiring  users  to  repeat  all  vocabulary  words  and 
phrases  numerous  times  before  ever  using  the  system  have  given  way  to  speaker  independent,  continuous 
speech  systems  requiring  no  prior  system  training.  Small,  static  vocabularies  of  300  or  less  words  have  been 
replaced  by  very  large,  dynamic  vocabularies  containing  tens  of  thousands  of  words  and  are  robust  enough  to 
handle  both  male  and  female  native  and  non-native  speakers.  The  systems  have  also  matured  to  the  point 
where  they  can  achieve  high  recognition  rates  in  very  noisy  environments  [14,15].  As  a  result  of  these  recent 
technology  advances,  numerous  military  applications  for  using  this  natural  and  intuitive  communication 
method  are  being  pursued  [16,17,18,19,20]. 

6.4.2.2  Actual  or  Potential  Applications  to  UMVs 

Numerous  laboratory  and  field  studies  have  shown  that  the  intelligent  use  of  speech  recognition  and  synthesis 
technology  can  significantly  improve  the  operator  interface.  In  an  aircraft  simulation  experiment  using  DVI  to 
control  single-seat  cockpit  functions  [10],  speech  was  found  to  be  a  viable  control  method  for  changing  radio 
frequencies  and  navigation  coordinates  and  for  designating  ground  targets  in  an  air  attack  mission.  Using  this 
technology,  data  entry  task  performance  was  increased,  while  overall  pilot  workload  was  reduced. 
When  multi-tasking,  pilots  were  able  to  enter  numeric  data  into  the  onboard  systems  while  their  hands  and 
eyes  were  busy.  In  addition,  the  interaction  of  using  speech  technology  combined  with  an  automatic  target 
cuing  system  resulted  in  superior  performance  in  selecting  and  marking  targets. 

In  a  study  designed  to  compare  data  entry  performance  using  speech  recognition  with  manual  (keyboard, 
mouse,  push  button)  methods  [17],  pilots  performed  data  entry  and  query  tasks  while  flying  a  simulated 
Unmanned  Aerial  Vehicle  mission.  The  operators  were  asked  to  perform  a  continuous  flight/navigation 
control  task  while  responding  to  intermittent  data  entry,  data  query  and  emergency  checklist  tasks.  The  results 
showed  that  data  entry  task  performance  improved  approximately  40%  when  pilots  used  DVI,  in  comparison 
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to  performance  with  conventional  manual  controls.  The  operators  also  judged  the  speech  system  to  be  easier 
to  use  than  manual  controls  and  resulting  in  lower  workload. 

The  U.S.  Army  has  also  investigated  the  use  of  speech  recognition  technology  for  ground  vehicle  operator 
interfaces  in  a  M1A2  tank  simulator  [21].  Results  showed  that  tank  commanders  could  perform  tasks  (such  as 
calls  for  fire,  medical  evacuation  requests  and  radio  setup)  faster  using  speech  input  compared  to  using 
conventional  push  buttons  and  a  joystick  cursor  controller.  Operators  could  eliminate  the  menu  “drill-down” 
search  for  the  “call  for  fire”  menu  function  by  simply  speaking  a  “call  for  fire”  command.  Operators  also 
reported  higher  perceived  levels  of  workload  when  using  the  conventional  input  method. 

Quite  a  bit  of  research  in  the  potential  uses  for  speech  recognition  technology  in  helicopters  has  been  done  in 
the  United  States  [22,23,24],  United  Kingdom  [16]  and  Canada  [25,26]  including  a  successful  flight  test  of  an 
ITT  system  in  the  mid  and  late  1980s.  Successful  applications  of  speech  technology  in  a  number  of  ground 
based  helicopter  simulation  platforms  as  well  as  a  number  of  actual  helicopter  flight  tests  in  the  United 
Kingdom,  Canada  and  the  United  States  are  described  in  [16,26]. 

Interesting  uses  for  speech  recognition  technology  for  training  systems  have  also  been  fielded  by  the  U.S. 
Naval  Air  Warfare  Center  Training  Systems  Division  (NAWCTSD)  in  Orlando,  Florida.  Some  systems 
already  in  operational  use  include: 

•  A  Carrier  Air  Traffic  Control  Center  (CATCC)  laboratory  which  realistically  simulates  the  at-sea  air 
traffic  control  environment.  A  speech  recognition  system  replaces  human  pilot  role  players  at  the 
CATCC  “C  School”,  an  advanced  technical  training  facility  in  Pensacola,  Florida  that  serves  90  Navy 
students  per  year  plus  team  training  for  each  of  the  Navy’s  carriers. 

•  A  Tower  Operated  Training  System  (TOTS)  laboratory,  also  in  Pensacola,  Florida,  serves  as  a 
VFR  Tower  Air  Traffic  Control  facility  trainer  that  simulates  the  visual  environment  of  a  tower  cab. 
TOTS  features  speech  recognition/synthesis  to  enable  direct  student  interaction  with  simulated  pilots. 

•  The  Navy’s  Advanced  Radar  Air  Traffic  Control  (ARATC)  trainer  in  Pensacola,  Florida,  realistically 
simulates  shore-based  air  search  and  ATC  radar  systems  using  speech  recognition/synthesis 
technology.  The  trainer  provides  for  approach,  arrival  and  precision  radar  final  approach  training  by 
allowing  students  to  communicate  with  simulated  pilots. 

•  The  Radar  Air  Traffic  Control  Labs  (Radlabs)  training  system  in  Pensacola  uses  speech  recognition  to 
teach  basic  radar  air  traffic  control  procedures. 

•  The  Amphibious  Air  Traffic  Control  Center  (AATCC),  a  realistic  trainer  for  the  amphibious  assault 
ship  environment,  uses  speech  recognition/synthesis  to  provide  realistic  interaction  with  the  simulated 
pilots. 

•  A  Landing  Signal  Officer  (LSO)  Trainer  at  Naval  Air  Station,  Oceana,  Virginia  features  a  wide  field- 
of-view  projection  of  a  carrier  landing  platform  and  allows  the  LSO  student  to  communicate  with  the 
pseudo-pilot  via  a  speech  recognition/synthesis  system. 

•  The  Air  Traffic  Control  Proficiency  Trainer  (APTS)  is  a  table-top  low  cost  trainer  developed  to  train 
basic  aircraft  control  and  phraseology.  APTS  recently  was  delivered  to  46  U.S.  Navy  and  Marine 
Corps  sites  worldwide. 

It  is  clear  from  the  number  of  successful  applications  of  this  technology  in  both  simulated  and  operational 
ground  and  flying  platforms  that  speech  recognition  will  be  an  important  and  integral  component  in  future 
military  operator  interfaces.  Indeed,  operational  use  of  the  technology  has  already  found  its  way  into  military 
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jet  aircraft.  In  Europe’s  newest  jet  fighter  aircraft,  the  Typhoon,  DVI  gives  pilots  the  ability  to  control  data 
entry  functions  such  as  changing  radio  frequency  and  displaying  fuel  status  as  well  as  more  critical  functions 
such  as  configuring  radars  and  radios,  all  without  having  to  move  their  hands  from  the  flight  controls. 
This  DVI  functionality  serves  as  an  alternative  to  using  manual  control  methods  when  the  pilot’s  hands  and 
eyes  are  busy. 

Following  the  Typhoon’s  lead,  the  new  French  Rafale  aircraft  will  have  a  DVI  capability,  allowing  the  pilot  to 
perform  actions  through  spoken  commands  using  a  vocabulary  of  90  to  300  words.  The  United  States  also 
plans  to  field  a  DVI  capability  in  its  new  Joint  Strike  Fighter  (JSF).  Under  a  Memorandum  of  Agreement 
signed  with  the  JSF  prime  contractor  Fockheed  Martin,  Adacel  (Melbourne,  Australia-based)  will  develop  a 
speech-enabled  cockpit  control  system  as  part  of  the  10-year  System  Development  and  Demonstration  phase 
of  the  JSF  program. 

Through  the  breadth  of  DVI  research  conducted  since  the  1950’s,  a  number  of  “lessons  learned”  have 
emerged  that  serve,  in  part,  as  suggested  guidelines  for  the  successful  implementation  of  this  technology. 
These  lessons,  together  with  general  guidelines  published  by  Gardner-Bonneau,  Rudnicky  and  others  for 
telephony  and  other  spoken  language  systems  [13,27]  should  serve  as  a  baseline  set  of  guidelines  for  the 
successful  implementation  of  speech  recognition  technology  in  military  systems.  Some  of  these  include: 

•  Fimit  or  eliminate  speaker  training  -  State  of  the  art  speech  recognition  systems  no  longer  require 
speakers  to  “train”  the  system  to  recognize  their  voice  commands.  Modem  DVI  systems 
automatically  adapt  to  the  ambient  acoustical  environment  and  the  nuances  of  a  speaker’s  voice  over 
time. 

•  Phrases  that  are  from  three  words  or  longer  are  more  distinctive  and  will  be  better  recognized  than 
short  one  or  two  word  phrases.  Military  operators  are  trained  to  use  short,  terse  speech  on  radios, 
but  single-word  commands  are  less  distinctive  and  can  be  confused  by  the  recognizer. 

•  Avoid  commands  that  sound  similar,  but  have  different  meanings.  For  example,  “Turn  backups  on” 
and  “Turn  backups  off’  differ  by  only  one  phoneme  and  might  be  confused  by  the  recognizer  in  a 
noisy  environment. 

•  Define  commands  that  are  easy  to  remember,  feel  natural  to  say,  and  don’t  conflict  with  menu  items 
or  controls. 

•  Make  use  of  “Macro”  commands.  Provide  speech-recognition  commands  that  add  value  by  doing 
more  than  can  be  accomplished  through  a  single  click  or  keyboard  equivalent. 

•  Use  confirmation  when  necessary.  For  commands  that  could  result  in  data  loss,  ask  for  confirmation 
before  executing  the  command. 

•  Redundancy  -  Make  every  attempt  to  provide  an  alternative  method  for  accomplishing  an  action. 
Don’t  make  speech  the  only  way  for  the  user  to  accomplish  a  task. 

•  Errors  will  occur.  Bias  toward  deletion  errors,  rather  than  errors  of  insertion  or  substitution. 

•  Have  a  clear  understanding  of  the  trade  offs  associated  with  using  speech  controls  vs.  manual  controls 
and  limit  the  use  of  speech  for  critical  tasks  that  could  be  accomplished  with  a  simple  button  push. 

•  Provide  feedback  to  the  user  to  indicate  that  the  speech  system  either  took  or  didn’t  take  action. 

•  If  graphic  feedback  is  impractical,  provide  auditory  feedback. 

•  At  all  times,  the  user  should  know  what  can  be  spoken. 
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•  Do  not  overwhelm  the  user  with  feedback. 

•  A  robust,  continuous  listening  system  is  ultimately  better  than  one  that  requires  explicit  user  action. 
Unfortunately,  in  many  military  systems,  due  to  high  ambient  background  noise  and/or  other  speakers 
in  close  proximity,  a  push-to-talk  implementation  is  probably  the  best  solution  (push  and  hold  a 
button  to  speak,  release  when  done  speaking). 

•  Offer  an  Undo  capability.  If  Undo  is  not  practical,  use  a  confirmation  protocol  in  cases  where  undoing 
a  recognition  error  is  troublesome. 

•  More  complex  applications  need  to  track  state,  so  that  actions  are  interpreted  in  the  proper  context. 

•  Out-of-Phraseology  Speech  -  Speech  recognizers  are  not  yet  capable  of  understanding  unconstrained 
human  speech.  Accordingly,  applications  are  developed  based  on  constrained  vocabularies  -  but  users 
often  say  words  that  are  not  in  the  legal  vocabulary.  Since  the  speech  recognizer  will  try  to  make  the 
best  match,  undetected  out-of-phraseology  speech  could  be  processed  with  chaotic  results. 
The  challenge  is  to  detect  out-of-phraseology  speech  and  reject  it  before  it  is  post-processed. 

•  If  a  certain  action  is  made  available  through  speech,  make  sure  that  tasks  involving  this  action  can  be 
completed  via  speech.  If  confirmation  is  required,  make  sure  the  user  can  speak  the  response  (instead 
of  forcing  the  user  to  type  something,  for  example).  Users  prefer  to  stay  in  one  input  mode  rather  than 
switching  back  and  forth  depending  on  the  task. 

6.4.2.3  Technology  Maturity,  Challenges,  and  Unresolved  Issues 

Speech  recognition  and  synthesis  technology  has  already  been  fielded  in  military  training  and  simulation 
systems  using  relatively  small  vocabularies.  System  prototypes  have  also  been  successfully  demonstrated  in 
operational  environments  such  as  aircraft  cockpits  and  ground  control  stations,  and  are  considered  to  be  at 
TRL  Rating  #7.  However,  the  ultimate  goal  of  speech  technology  researchers  is  to  create  a  system  that  listens, 
understands  and  behaves  as  well  as  (or  better  than)  a  human  might  in  the  same  circumstances.  Current  state  of 
the  art  systems  are  quite  good  at  listening  for  predefined  words  and  phrases,  but  are  not  yet  very  good  at 
understanding  unconstrained,  natural  language  speech.  Significant  research  in  machine  language 
understanding  is  currently  being  done  for  both  commercial  and  military  systems.  The  U.S.  Defense  Advanced 
Research  Projects  Agency  (DARPA)  has  funded  much  of  this  work  under  their  Communicator  and  other 
programs.  Several  leading  universities  have  built  variations  of  the  Communicator  architecture,  including 
MIT’s  Galaxy  Communicator  system  [28],  Carnegie  Mellon  University’s  (CMU)  Communicator  system  [29], 
and  the  University  of  Colorado’s  CU  Communicator  system  [30]. 
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6.4.3  Additional  Alternative  Input  Devices 

Control  devices  that  require  a  manual  input  impose  restrictions  on  the  operator’s  posture  at  the  control  station. 
Moreover,  often  they  require  the  operator’s  gaze  be  directed  at  the  control  device  and  keep  the  operator’s 
hands  busy.  There  are  numerous  alternative  controls  that  have  the  potential  of  providing  novel  means  of 
interacting  with  UMV  systems  (e.g.,  hands-free,  head-up  input).  Speech  recognition  technology,  described  in 
detail  above,  is  one  such  technology.  Non-conventional  controllers  that  do  not  require  a  direct  mechanical 
linkage  between  the  operator  and  the  input  device  are  desirable  for  supplementing  manual  control. 
These  alternative  controls  use  signals  from  the  brain,  muscles,  voice,  lips,  head  position,  eye  position, 
and  gestures.  Some  of  these  alternative  controls  are  not  yet  commercially  available  or  configured  to  facilitate 
integration  with  software  under  development  for  supervisory  ground  control  station  applications. 
Plus,  the  control  achieved  with  many  hands-free  devices  can  be  described  as  rudimentary.  Any  UMV 
application  must  take  into  account  the  respective  dimensionality,  accuracy,  speed  and  bandwidth  of  control 
afforded  by  each  alternative  input  device.  Of  these  alternative  input  devices,  speech  recognition  is  the  only 
technology  with  sufficient  maturity  and  reasonable  cost  to  be  considered  a  near-term  candidate  for  UMV 
control  station  application  (TRL  Rating  #7).  Since  speech  recognition  technology  was  described  earlier,  this 
section  includes  brief  overviews  of  the  other  alternative  input  devices  whose  Technology  Readiness  Levels 
range  from  1  (EEG-based  control)  to  4  (gesture  and  eye-based  control). 

6.4.3. 1  Gesture-Based  Control 

Hand  and  body  gestures  are  an  important  component  of  communication.  Gesture-based  control  seeks  to 
exploit  this  channel  for  system  control.  The  head,  hands  and  body  can  be  localized  in  space  using  trackers, 
video  techniques,  gloves  or  suits.  Trackers  (mechanical,  electromagnetic,  ultrasonic,  and  optical)  are  devices 
that  allow  one  to  measure  the  position  and  orientation  of  a  body  part  in  space  [1].  Video  techniques  use  image 
processing  in  order  to  identify  a  specific  body  part  and  then  reconstruct  its  position,  orientation  and  posture 
from  two-dimensional  video  images  [2].  Gloves  and  suits  allow  one  to  measure  the  relative  positions  and 
angles  of  body  components.  A  directory  of  manufacturers  of  gesture  capture  devices  is  available  [3].  Perhaps 
the  most  highly  developed  for  system  interactions  are  devices  that  employ  glove-based  electronic  input. 
For  applications  in  which  the  body  and  hands  are  involved  in  other  activities,  gesture-based  control  for  hands¬ 
free  operation  may  best  involve  detecting  defined  movements  or  positions  of  the  operator’s  face  or  lips. 
For  instance,  optical  and  ultrasonic  sensing  technologies  have  been  used  to  monitor  an  operator’s  mouth 
movement  [4].  Two  candidate  control  approaches  include  processing  lip  movements  to  provide  Tip  reading’ 
and  translating  symbolic  lip  gestures  into  control  inputs  [5]. 

One  challenge  with  gesture-based  control  is  identifying  the  beginning  and  end  points  of  a  gesture  [1].  Typical 
solutions  require  the  operator  to  take  a  ‘default’  posture  between  gestures,  which  serves  as  an  anchor  for  the 
system.  Another  problem  is  detecting  whether  a  gesture  is  addressed  to  the  recognition  system  or  is  a  part  of 
normal  interpersonal  communication.  Creating  an  active  zone  in  which  gestures  are  effective  is  one  solution. 
Given  that  a  specific  gesture  can  be  reliably  discriminated  from  other  activity,  there  are  three  styles  of  input 
that  can  be  made  with  gestures  [1].  With  “direct  input”,  the  features  of  the  gesture  generate  kinematically 
similar  actions  in  the  task  domain  (e.g.,  one-to-one  control  of  a  robot  hand).  With  “mapped  input”,  features  of 
the  gesture  are  mapped  in  some  logical  fashion  to  actions  in  the  task  domain  (e.g.,  number  of  raised  fingers 
indicating  a  parameter  level).  For  “symbolic  inputs”,  features  of  the  gesture  are  interpreted  as  commands  to 
the  system.  Most  uses  of  facial  expressions  would  fall  into  the  latter  category. 

A  potential  UMV  control  application  is  illustrated  by  a  study  that  examined  continuous  cursor  controllers  to 
designate  targets  in  a  stereoscopic  three-dimensional  tactical  map  resident  in  a  manned  aircraft  station 
simulator  [6].  The  tracking  volume  was  remote  from  the  actual  map  so  that  hand  movements  were  actually 
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made  in  a  space  close  to  the  aircraft  controls  rather  than  within  the  volume  of  the  map.  This  hand  volume  was 
reduced  in  scale  so  that  hand  movements  were  small  compared  with  the  size  of  the  map.  Results  showed  that 
target  designation  was  faster  with  an  ultrasonic  hand  tracker  compared  to  designation  with  a  four- axis  joystick 
and  a  voice  control  system. 


6.4.3.2  Gaze-Based  Control 

Operators  naturally  look  at  objects  that  they  want  to  manipulate  or  use.  For  applications  in  which  the  operator 
views  a  display  during  control  operations,  harnessing  the  direction  of  gaze  promises  to  be  a  very  natural  and 
efficient  control  interface.  With  such  control,  the  computer  initiates  a  predefined  action  once  it  receives  an 
input  based  on  the  operator’s  point-of-gaze  [7].  For  example,  gaze-based  control  could  be  used  to  designate 
points  of  interest  on  a  display  showing  the  view  from  a  UMV-mounted  camera.  In  addition  to  command-and- 
control  applications,  eye  gaze  can  be  used  to  detect  visual  attention  and  adapt  the  application  based  on  this 
information  [8].  For  UMV  applications,  an  intelligent  decision  aiding  system  might  adapt  the  display  format 
or  tailor  active  speech  commands  based  on  the  operator’s  point  of  gaze. 

Unless  the  head  is  stationary  (or,  with  some  systems,  held  within  a  small  motion  box),  determining  eye  gaze 
point  also  involves  tracking  the  head  position/rotation.  Predominant  eye  tracking  techniques  can  be  classified 
as  electro-oculographic,  scleral  coil,  and  optical  methods.  The  optical  classification  is  the  largest  of  these 
categories  and  several  methods  within  this  category  are  the  most  commonly  used  for  gaze-based  control 
(e.g.,  corneal  reflection  trackers)  [9].  The  most  frequent  problem  with  gaze-based  control  is  avoiding  what 
Jacob  [10]  dubbed  the  “Midas  touch”  problem,  with  commands  activating  where  ever  the  operator  gazes. 
Careful  interface  design  is  also  necessary  to  ensure  that  the  operator’s  head  and/or  eye  movements  during  task 
completion  are  natural  and  not  fatiguing.  Rather,  natural  head  and  eye  movements  should  be  used  to  provide  a 
direct  pointing  capability  that  can  be  combined  with  other  control  modalities  to  command  activation. 
Plus,  the  control  design  needs  to  take  into  account  the  accuracy  limits  of  the  gaze  measurement  system  and  its 
operation  under  variable  illumination  environments  [9]. 


6.4.3.3  Electromyographic  (EMG)-Based  Control 

EMG-based  control  uses  the  electrical  signals  that  accompany  muscle  contractions,  rather  than  the  movement 
produced  by  these  contractions,  for  control.  Electrodes  positioned  on  the  surface  of  the  skin  detect  these 
electrical  signals  (e.g.,  produced  during  an  operator’s  clenching  of  the  jaw).  Next,  the  signal’s  parameters  are 
translated  into  a  control  input  with  processing.  The  simplest  algorithm  employs  an  on-off  control  based  on  the 
level  or  rate  of  change  of  EMG  activity.  For  example,  if  muscle  activity  at  one  electrode  site  exceeds  some 
threshold,  a  prosthetic  hand  opens.  Above-threshold  activity  at  another  site  causes  the  hand  to  close.  Control 
can  also  be  based  on  EMG  patterns  rather  than  on  levels  or  rates  of  change  [11]. 

EMG-base  control  is  also  a  potential  input  device  for  computer  systems.  Research  has  shown  that  signals 
extracted  from  electrodes  on  the  forehead  can  be  used  to  control  the  movement  of  a  cursor  to  track  computer¬ 
generated  targets,  as  well  as  provide  a  quick,  accurate  discrete  on/off  response  [12].  Thus,  for  UMV 
applications,  signals  from  electrodes  positioned  on  the  operator’s  face  may  provide  an  alternative  hands-free, 
head-up  binary  input  response.  Continued  development  in  EMG-based  control  is  required  to  optimize  the 
signals  employed,  assess  the  stability  of  the  electrode  contact  over  time,  and  minimize  the  effect  of  operator 
movement  and  external  electrical  activity  on  signal  recordings  [11]. 


6-18 


RTO-TR-HFM-078 


dMNATO 
WP  I  OTAN 


ADVANCED  UMV  OPERATOR  INTERFACES 


6.4.3 A  Electroencephalographic  (EEG)-Based  Control 

Electrodes  positioned  on  the  surface  of  the  scalp  can  record  signals  that  represent  a  summation  of  the 
electrical  activity  of  the  brain.  Although  much  of  the  recorded  EEG  appears  to  be  noise-like,  it  does  contain 
specific  rhythms  and  patterns  that  represent  the  synchronized  activity  of  large  groups  of  neurons.  There  are 
two  general  approaches  for  translating  the  electrical  activity  of  the  brain  into  a  control  signal.  In  one  approach, 
EEG  patterns  are  brought  under  conscious  voluntary  control  with  training  and  biofeedback.  For  example, 
voluntarily  increasing  the  EEG  activity  in  a  specific  frequency  band  above  a  threshold  might  be  used  to  turn  a 
switch  on.  Another  approach  harnesses  involuntary,  naturally  occurring  brain  rhythms,  patterns,  and  responses 
that  correspond  to  human  sensory  processing,  cognitive  activity  or  motor  control.  For  example,  an  operator 
might  imagine  moving  their  right  hand  to  push  a  button.  The  computer  would  recognize  the  EEG  pattern 
associated  with  this  movement  preparation  and  operate  the  right-hand  button  without  further  operator  action. 
Although  detection  of  these  signals  is  easily  accomplished  with  inexpensive  components,  optimization  of  this 
alternative  control  requires  minimizing  the  time  required  for  signal  processing  and  developing  easily  donned 
electrodes  [11,13,14]. 

EEG-based  control  has  been  demonstrated  for  several  applications:  cursor  control,  alphanumeric  input,  binary 
operation  of  a  neuromuscular  stimulator,  and  roll-axis  control  of  a  motion  simulator  [14,15].  To  date, 
only  rudimentary  control  has  been  achieved.  However,  the  ability  to  make  a  system  input,  albeit  crude, 
without  lifting  a  finger,  uttering  a  sound,  or  directing  one’s  gaze,  inspires  the  dream  of  someday  enabling  a 
genuine  “thought-based  interface”. 


6.4.3.5  Multi-Modal  Input  Design 

For  both  manual  and  alternative  controls,  in  addition  to  considering  the  general  adequacy  of  the  candidate 
control,  the  specific  mapping  of  the  input  device  to  control  tasks  must  be  addressed.  It  is  unlikely  that  a  single 
input  device  will  be  adequate  for  all  the  control  functions  required  in  a  particular  application.  A  specific  input 
device  will  be  elegantly  appropriate  for  some  control  tasks  and  clearly  inappropriate  for  others.  For  example, 
speech-based  control  can  be  useful  for  a  variety  of  control  tasks,  but  use  of  a  speech  command  to  designate  a 
position  on  a  two-dimensional  surface  can  be  cumbersome.  In  contrast,  gaze-based  control  is  efficient  for 
designating  a  position  on  a  map  display,  but  an  auxiliary  control  modality  is  more  useful  for  commanding  the 
system  to  act  on  the  designated  location.  This  action  command  might  be  a  voice  utterance  [16]  or  voluntary 
facial  muscle  activation  [17].  The  use  of  multiple  control  input  devices  capitalizes  on  using  voluntary  gaze 
direction  to  rapidly  designate  a  position  and  a  voice  utterance  or  voluntary  facial  muscle  activation  (EMG) 
commands  to  quickly  initiate  an  action.  A  multi-modal  input  method  that  combines  speech  and  touch  screen 
operation  was  also  found  to  create  a  more  operator-friendly  interface  [18].  Thus,  task-controller  mapping  must 
take  into  account  how  best  to  increase  overall  functionality  by  using  multiple  input  devices. 


6.4.3.6  References  for  Additional  Alternative  Input  Devices 

[1]  McMillan,  G.R.  and  Calhoun,  G.L.  (2000).  Gesture -based  Control,  International  Encyclopedia  of 
Ergonomics  and  Human  Factors,  New  York:  Taylor  and  Francis,  Inc.,  Volume  1:237-239. 

[2]  Borghi,  F.,  Lombardi,  L.  and  Porta,  M.  (2005).  Basic  hand  gesture  recognition  for  human-computer 
communication.  Proceedings  of  the  11th  International  Conference  on  Human-computer  Interaction, 
CD-ROM. 

[3]  Buxton,  B.  (2005).  A  directory  of  sources  for  input  technologies.  Available  at  http://www.billbuxton. 
com/InputSources.html 
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6.5  DATA  DISPLAY  TECHNOLOGIES 

6.5.1  Head-Mounted  Displays 

6.5.1. 1  Description  of  Technology 

A  head-mounted  display  (HMD)  presents  real  video  imagery  or  synthetically  generated  visual  imagery  via  a 
head-mounted  optic  system  with  very  small  displays  attached  to  a  helmet,  visor,  or  set  of  spectacles 
(see  Figure  6-3).  There  is  a  wide  range  of  technologies  and  approaches  associated  with  today’s  HMD  systems, 
including  the  type  and  quality  of  image  display,  monocular  versus  bi-ocular  design,  the  ability  to  display 
colour  images,  the  ability  to  effectively  display  stereoscopic  three  dimensional  (3D)  images,  system 
size/weight,  and  the  ability  to  concurrently  view  the  local  external  environment.  As  this  section  is  intended 
primarily  as  a  short  summary  of  the  HMD  technology  and  its  relevance  to  UMVs,  interested  readers  should 
refer  to  [1,2,3]  for  more  comprehensive  coverage  of  this  area. 


Figure  6-3:  Examples  of  HMDs. 

HMDs  can  support  either  immersive  or  augmented  reality  applications,  depending  upon  the  transparency  of 
the  head-mounted  optics.  Immersive  HMDs  require  the  user  to  view  only  the  image  presented  via  the  HMD 
optic  system,  while  augmented  reality  HMDs  allow  the  user  to  “see-through”  the  HMD  display, 
thus  combining  imagery  from  the  HMD  with  the  surrounding  real-world  visual  field.  This  section  only 
considers  immersive  HMD  display  technology,  i.e.,  designs  that  occlude  the  subject’s  view  of  his/her 
immediately  surrounding  physical  environment.  Augmented  reality  technology  is  discussed  in  Section  6.5.2. 

This  section  also  only  considers  head-coupled  HMD  applications  (i.e.,  the  HMD  visual  image  is  updated  in 
response  to  head  movements,  via  a  position  tracking  sensor  that  provides  the  computer  with  the  current  head 
location/orientation  information).  Head-coupled  HMDs  enable  a  synthetically  generated  visual  scene  to  be 
continually  modified  in  response  to  head  movements  so  that,  no  matter  how  the  user  moves,  objects  in  the 
viewed  scene  appear  to  remain  in  stable  locations  (thus  providing  the  impression  that  the  user  is  moving 
within  the  virtually  generated  space).  Alternatively,  in  tele-operated  robotic  systems,  the  motion  of  the  user’s 
head  can  be  used  to  control  the  position/orientation  of  a  remote  camera  (or  other  robotic  action)  [4]. 
Head-coupled  HMDs  offer  the  highest  potential  degree  of  immersiveness  and  utility.  However  there  remain 
many  limitations  due  to  existing  technology,  as  will  be  discussed  below. 
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Depending  upon  the  particular  UMV  application,  it  may  be  critical  for  a  HMD  to  convey  accurate  depth 
information  in  an  intuitive  and  accurate  manner.  Although  all  HMDs  can  convey  a  sense  of  depth  and  distance 
using  conventional  two-dimensional  depth  cues  (including  linear  perspective,  interposition,  relative  size, 
texture  gradients,  etc.),  certain  HMDs  also  have  the  potential  to  portray  depth  via  various  stereoscopic 
techniques.  Stereopsis  can  provide  an  intuitive,  unambiguous  cue  for  depth  and  it  dominates  most  other  depth 
cues.  Dichoptic  presentation  involves  using  two  monitors  to  portray  a  scene,  one  monitor  per  eye,  each  with 
its  appropriate  viewpoint  for  that  eye’s  position  in  space  [5].  This  method  utilizes  binocular  fusion  to  yield 
stereopsis.  Electronic  shutter  glasses  use  one  monitor  to  present  a  stereoscopic  image  by  providing  two 
alternating  views  of  a  scene  (corresponding  to  the  viewpoint  disparity  between  the  two  eyes),  synchronized  to 
the  frame  rate,  such  that  one  interleaved  frame  in  each  pair  is  presented  to  each  eye.  Section  6.5.3  contains 
more  information  on  various  3D  display  technologies. 


6.5.1.2  Actual  or  Potential  Application  to  UMVs 

HMDs  have  been  found  to  enhance  wide-area  search  and  intercept  operations  performed  by  manned  aircraft 
[6].  A  potential  advantage  of  head-coupled  control  versus  manual  control  over  one’s  viewpoint  is  the  addition 
of  ecologically  relevant  proprioceptive  cues  which  provide  motion  information  based  on  vestibular  inputs, 
joint  angles,  muscle  lengths,  and  tendon  tension  during  head  movements.  Head-coupled  HMDs  are  also 
hypothesized  to  reduce  cognitive  processing  demands  in  achieving  new  viewpoints.  Some  studies  have 
suggested  that  head-coupled  configurations  facilitate  awareness  of  areas  already  searched,  thereby  potentially 
reducing  the  re-scanning  of  those  same  areas  [7].  Thus,  HMD  technology  may  benefit  UMV  operators, 
especially  since  reduced  situation  awareness  is  often  a  by-product  of  current  UMV  control  station  designs 
[8,9].  It  is  theorized  that  a  HMD  could  enhance  the  operator’s  large-area  searches  and  overall  spatial 
orientation  of  the  remote  environment,  while  larger,  high-resolution  fixed  console  displays  could  be  reserved 
for  any  target  fine  discrimination  tasks.  Other  expected  advantages  of  HMDs  include  hands-free  control  and 
intuitive  operation.  However,  studies  investigating  the  benefits  of  HMDs  have  so  far  produced  mixed  results. 
Below  is  a  summary  of  the  recent  research  in  the  area,  categorized  by  type  of  UMV:  UAV,  UGV  and  UUV. 

6.5. 1.2.1  UAVs 

HMDs  have  been  demonstrated  to  have  a  positive  impact  of  certain  UAV  control  tasks.  A  study  [10]  explored 
UAV  operator  control  of  a  remotely-operated  helicopter  using  an  omni-directional  camera  controlled  by  a 
head-coupled  HMD.  The  HMD  system  was  found  to  promote  operator  immersion,  encouraging  a  feeling  of 
‘presence’  as  though  the  operator  was  physically  in  the  vehicle.  The  HMD  also  resulted  in  faster  and  more 
accurate  completion  times  in  a  simulated  helicopter  control  task,  compared  to  the  alternative  of  attempting  to 
control  the  helicopter  while  viewing  it  directly  from  the  ground.  These  results  support  claims  that  HMDs  can 
provide  increased  situation  awareness.  However,  the  non-HMD  condition  was  somewhat  lacking  in  that  it  did 
not  include  a  fixed-display  out-the-window  view  from  the  helicopter,  so  it  is  unclear  whether  viewpoint 
location  or  head-coupled  HMD  provided  the  observed  improvements.  Another  study  [11]  explored  HMDs  for 
small  UAV  applications.  The  task  involved  piloting  a  small  UAV  past  a  ground  target  and  then  turning  around 
at  various  distances  to  re-acquire  that  same  ground  target.  The  researchers  found  that  the  use  of  a  head- 
coupled  HMD  resulted  in  faster  and  more  successful  re-acquisition  of  the  ground  target  than  when  using  a 
conventional  display  of  imagery  from  the  UAV’s  nose-camera.  However  the  horizontal  field-of  view  was 
nearly  twice  as  large  for  the  HMD  as  compared  to  the  conventional  display,  which  may  have  contributed  to 
these  findings.  Additionally,  all  subjects  complained  of  discomfort  when  wearing  the  HMD.  Nonetheless, 
there  is  research  support  for  the  proposition  that  HMDs  can  provide  greater  situation  awareness  resulting  in 
increased  UAV  operator  performance. 
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An  early  experiment  at  the  TNO  Research  Institute  was  conducted  to  explore  the  applicability  of  a 
head-slaved  camera  system  in  UMV  applications  [12].  To  overcome  some  possible  drawbacks  of  HMDs 
(e.g.,  weight),  a  HMD  was  compared  with  a  head-slaved  dome  projection  (Figure  6-4).  To  overcome  the 
possible  drawbacks  of  transmission  delay,  a  method  was  introduced  to  compensate  for  the  spatial  distortions. 
This  technique,  called  delay  handling,  preserves  the  correct  spatial  relation  between  the  viewing  direction  of 
the  camera  and  the  operator,  by  presenting  incoming  images  in  the  camera  viewing  direction  at  the  moment 
the  images  were  recorded,  and  not  in  the  actual  viewing  direction  of  the  operator.  The  results  showed  that 
delay  handling  is  successful  in  supporting  the  perception  of  correct  spatial  relations.  No  differences  in  task 
performance  were  found  between  the  actual  HMD  and  the  dome  projection. 


Figure  6-4:  Dome  Projection  in  which  the  Camera  Direction  is  Head-Coupled,  and  the  Operator 
Receives  High  Quality  Proprioceptive  Feedback  on  Camera  Viewing  Direction. 


In  follow-on  studies  at  TNO,  researchers  compared  operator  performance  with  head-coupled  camera  control, 
and  HMDs  with  manual  camera  control  [13,14].  Subjects  had  to  locate  targets  as  fast  as  possible.  The  results 
showed  that  head-slaved  camera  control  increased  search  speed,  but  enlarged  the  search  path  as  compared  to 
manual  (joystick)  control.  An  increased  susceptibility,  during  head-slaved  control,  to  mismatches  between 
visual  information  and  proprioceptive  information  may  account  for  these  findings.  Additional  measures  of 
head  movements  showed  that  eye-head  coordination  was  altered  during  head-slaved  camera  control.  Since  in 
these  experiments,  proprioceptive  feedback  was  available  in  the  manual  control  condition  as  well  (the  images 
were  presented  on  a  projection  screen  under  the  correct  camera  viewing  direction),  the  findings  imply  an 
additional  advantage  of  head-slaved  control  compared  with  manual  control  without  proprioceptive  feedback 
(as  would  be  the  case  when  using  a  fixed  monitor).  However,  [15]  found  that  employing  simulated  HMD 
images  projected  onto  a  large  screen  resulted  in  higher  UAV  operator  performance  than  when  they  used  an 
actual  HMD,  and  [16]  found  that  use  of  a  conventional  joystick  for  UAV  control  resulted  in  better 
performance  than  the  head-coupled  HMD.  These  latter  results  converge  with  several  other  studies,  as  detailed 
below. 

Two  experiments  were  conducted  at  the  U.S.  Air  Force  Research  Laboratory  to  evaluate  the  usefulness  of 
HMDs  for  UAV  tasks  involving  the  search  for  ground  targets  (Figure  6-5)  [17,18].  The  overall  approach  was 
to  compare  the  utility  of  a  manual  joystick  with  associated  stationary  display  monitor  (the  Baseline  Condition) 
to  that  of  various  head-coupled  HMD  configurations.  Specifically,  gimbal  camera  orientation  (azimuth  and 
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elevation  angle)  was  controlled  via  either  a  right-hand  control  stick  or  head-coupled  HMD,  while  camera 
zoom  was  always  controlled  with  the  left-hand  forward/aft  stick.  In  one  study  [17],  the  task  involved 
conducting  a  wide  area  search  followed  by  a  target  identification  task.  The  wide-area  search  was  conducted  by 
using  the  baseline  configuration  (control  stick  and  stationary  display  monitor)  or  a  head-coupled  HMD. 
The  target  identification  task  was  always  conducted  using  the  higher  resolution  stationary  display  monitor, 
as  the  HMD  did  not  have  the  required  resolution  to  afford  fine  discrimination.  Thus,  in  the  HMD  conditions, 
there  was  a  need  to  switch  displays  between  search  and  identification  tasks.  The  results  failed  to  show  any 
benefit  for  HMD-based  configurations.  Search  time  was  shorter  and  workload  was  lower  with  the  Baseline 
Condition  than  any  of  the  HMD  conditions.  Additionally,  many  subjects  experienced  discomfort  and 
simulator  sickness  symptoms  with  the  HMD  configuration. 


Figure  6-5:  UAV  Workstation  with  Head-Coupled  HMD 
and  Stationary  CRT  Camera  Displays. 

A  follow-on  study  [18]  was  conducted  to  specifically  evaluate  the  utility  of  a  head-coupled  HMD  for  the  SO’s 
conduct  of  a  360-degree  large  area  search  for  multiple  ground  targets.  This  study  did  not  include  the 
additional  target  identification  step  that  had  required  a  switch  from  HMD  to  a  stationary  display  in  the 
previous  study.  Six  camera  control/display  configurations  were  evaluated;  two  involved  the  stationary  display 
monitor  (each  with  a  different  rate  gain  joystick)  and  four  involved  a  HMD.  The  four  HMD  configurations 
varied  in  the  degree  to  which  the  camera  moved  with  head  movements.  One  “hybrid”  configuration  was  also 
evaluated  whereby  the  gimb ailed  camera  orientation  could  be  controlled  with  both  the  head  and  the  joystick 
simultaneously.  Results  indicated  fewer  unique  targets  were  prosecuted  with  the  HMD  than  with  the  fixed 
display  monitor.  Head-coupled  control  also  resulted  in  more  duplicate  target  designations,  higher  workload, 
and  lower  situation  awareness  ratings.  These  results  suggest  that  there  is  no  clear  advantage  for  head-coupled 
HMDs  in  the  performance  of  large-area  search  tasks.  In  fact,  performance  significantly  decreased  in  some 
experimental  manipulations  involving  the  HMD. 

A  similar  set  of  studies  were  conducted  utilizing  a  simulation  of  a  smaller  UAV  [19,20].  One  study  compared 
a  conventional  display  monitor  to  a  HMD  for  target  search,  discrimination  and  designation  tasks  [19]. 
Although  there  were  no  differences  between  display  conditions  for  target  detection  accuracy,  the  conventional 
display  condition  enabled  more  targets  to  be  correctly  identified  from  further  away  and  allowed  for  more 
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accurate  cursor  designation  of  those  targets.  Additionally,  subjects  experienced  far  more  discomfort 
(e.g.,  nausea,  disorientation,  eyestrain)  with  the  HMD  condition.  In  a  follow-on  study  [20],  these  researchers 
explored  the  effect  of  including  various  auditory  cues  (mono,  stereo,  3D  spatialized)  to  the  ground  target 
location  with  the  comparison  of  visual  display  conditions  (conventional,  head-coupled  HMD).  The  results 
confirmed  earlier  findings  that  conventional  displays  resulted  in  significantly  more  precise  target  designations 
and  fewer  reports  of  discomfort.  However,  although  HMD  conditions  yielded  higher  operator  workload 
ratings  than  conventional  displays  across  all  conditions,  3D  spatialized  cueing  reduced  HMD  workload  levels 
significantly. 

6. 5. 1.2.2  UGVs 

HMDs  have  characteristics  that  potentially  offer  many  advantages  over  conventional  UGV  operator  control 
units  [21].  Advantages  identified  included  the  system’s  light  weight,  decreased  power  consumption,  daylight 
readability,  and  theorized  improvements  in  operator  situation  awareness  and  telepresence.  HMDs  have  also 
been  demonstrated  in  UGV  systems  [22].  HMDs  were  found  to  be  beneficial  during  a  demonstration  of  the 
feasibility  of  utilizing  a  dune  buggy  as  a  UGV  travelling  complex  terrain  [23].  Other  researchers  conducted  a 
study  which  found  telepresence,  created  from  use  of  stereo  TV  imagery  displayed  in  an  HMD,  permitted 
operators’  to  drive  UGVs  at  higher  speeds  and  on  steeper  side  slopes  by  providing  an  enhanced  sense  of 
spatial/geographic  awareness  [24].  There  has  also  been  implementation  of  HMDs  into  operational  UGV 
systems.  Man-portable  UGVs  are  completing  missions  in  current  military  operations  with  operators  who  wear 
monocular  HMDs  [25].  Soldiers  are  successfully  controlling  the  UGVs  with  a  portable  joystick  and  HMD  to 
explore  cave  complexes  and  suspected  enemy  compounds.  Packbots’  success  in  combat  environments 
demonstrate  HMDs’  increasing  and  promising  role  in  UGV  control  station  design. 

6.5. 1.2.3  UUVs 

Although  few  formal  studies  have  been  conducted  in  this  area,  the  potential  value  of  HMDs  to  UUV  systems 
seems  promising  for  underwater  operations  and  operator  training  [26,27,37].  Other  researchers  have  described 
the  potential  importance  of  providing  UUV  operators  with  meaningful  cues  for  situation  awareness, 
good  workspace  visibility,  and  vehicle  behaviour  feedback  for  effective  performance  [28].  The  testbed  they 
designed  to  address  underwater  telerobotics  included  a  head-coupled  HMD  option.  Though  there  is  little 
research  specifically  addressing  HMDs’  effectiveness  compared  to  other  systems  in  UUV  operation, 
the  difficulties  for  operators  controlling  UAVs  and  UGVs  are  similar  to  those  in  underwater  vehicles  and  so  it 
is  reasonable  to  assume  that  research  on  HMDs  in  these  systems  could  transfer  to  UUVs. 


6.5.1.3  Technology  Maturity,  Challenges,  and  Unresolved  Issues 

HMDs  have  improved  considerably  since  they  were  first  introduced  into  the  commercial  market.  However, 
there  is  still  much  research  needed  before  they  achieve  widespread  appeal  in  military  and  consumer 
applications.  Research  areas  discussed  below  include  ergonomic  issues,  resolution,  time  latencies,  field-of- 
view,  and  the  occurrence  of  motion-sickness  type  symptoms. 

6. 5. 1.3.1  Ergonomic  Issues 

Ergonomic  issues  associated  with  HMDs  can  be  primarily  attributed  to  anthropometric,  biomechanic, 
and  psychomotor  concerns.  Most  HMDs  involve  some  encumbrance  by  the  user,  though  this  varies  with 
particular  equipment  chosen.  Lack  of  fit  is  a  primary  complaint  of  users  [29].  This  includes  inappropriate  fit, 
movement  limitations,  excessive  weight  and/or  size,  and  improper  distribution  of  the  weight.  HMD  weight 
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also  has  the  potential  to  alter  eye-head  coordination.  Suggestions  exist  for  improving  fit  [29].  Newer  displays 
are  being  developed  to  minimize  size  and  weight  such  that  they  can  be  clipped  onto  existing  eye-pieces, 
although  other  tradeoffs  exist  (limited  resolution,  small  field-of-view,  display  placement  within  larger  visual 
field,  etc.).  Furthermore,  certain  HMDs  enable  the  display  system  to  be  removed  or  rotated  out  of  the  way  to 
afford  intermittent  HMD  use  within  a  larger  real-world  work  task.  However  it  is  unknown  which  method  is 
most  preferable  for  various  UMV  applications.  Additional  information  regarding  ergonomic  issues  can  be 
found  elsewhere  [1,3,4].  Much  research  is  needed  to  improve  the  many  ergonomic  issues  of  HMDs. 

6.5. 1.3.2  Spatial  Resolution 

Spatial  resolution  is  a  measure  of  the  level  of  detail  available  in  a  visual  display  [2].  However,  it  can  be  a 
misrepresented  parameter  in  HMD  specifications.  Often,  resolution  is  described  in  terms  of  number  of  pixels 
in  a  display.  However,  the  size  of  the  display  and  its  distance  from  the  observer  also  contribute  to  the  effective 
resolution.  Increasing  display  field-of-view  reduces  effective  resolution  by  enlarging  each  pixel  in  the  same 
manner.  Therefore  a  more  effective  manner  in  which  to  specify  resolution  is  in  terms  of  visual  angle 
subtended  per  pixel,  termed  ‘angular  resolution’.  Angular  resolution  is  poor  in  most  current  HMDs,  far  lower 
than  the  resolving  capability  of  the  human  eye  (approximately  1  arc  min  visual  angle  or  less  [2]). 
Thus  researchers  have  found  resolution  to  be  a  limiting  factor  in  HMD  utilization  [30].  An  additional 
confusion  with  assessing  spatial  resolution  of  color  HMDs  is  associated  with  the  pixel-type  used  for 
determining  angular  resolution.  Color  display  pixels  are  often  formed  by  grouping  3  or  4  monocular  pixels  of 
different  wavelengths  (such  as  red,  green,  blue).  Display  manufacturers  sometimes  report  the  number  of  pixels 
and  angular  resolution  based  upon  the  total  number  of  monocular  pixels  available  instead  of  available  color 
pixels. 

Research  is  needed  to  better  define  spatial  resolution  requirements  of  HMDs  for  the  range  of  envisioned  UMV 
tasks.  Additionally,  research  is  needed  to  improve  spatial  resolution.  One  promising  technology  in  this  area  is 
the  virtual  retinal  display  [31]. 

6. 5. 1.3. 3  Time  Latencies 

Time  delays  exist  between  movements  made  by  the  user’s  head  (which  are  tracked  by  a  position-sensing 
device)  and  the  response  of  the  HMD  scene  to  those  movements,  due  to  delays  in  position  tracking  and  image 
generation  [4].  Time  delays  between  head  movements  and  virtual  image  response  result  in  loss  of  visual 
stability  which  can  affect  task  performance  and  generate  a  sensory  rearrangement  between  visual  and 
vestibular  cues  of  motion.  These  sensory  rearrangements  are  believed  to  induce  simulator  sickness  symptoms 
[32,33,34].  When  the  additional  time  delay  associated  with  UMV  datalink  communications  are  factored  in, 
the  total  delay  can  be  on  the  order  of  several  seconds.  Additionally,  time  delays  can  affect  user  acceptance  [7]. 

Specifications  are  needed  for  acceptable  HMD  time  delay  for  various  UMV  applications,  factoring  in  the 
delays  associated  with  communication  with  the  vehicle.  Acceptable  time  delays  for  UUV  operations  may  not 
be  acceptable  for  fast-moving  UAV  systems.  Research  is  also  needed  on  methods  to  minimize  the  negative 
effects  of  time  delay,  such  as  through  the  use  of  prediction  techniques  for  head  motion. 

6.5. 1.3.4  Display  Field-of-View  (DFOV) 

DFOV  is  the  visual  angle  subtended  by  the  display  screen  from  a  given  observer  location  [35]. 
This  parameter,  described  in  terms  of  its  horizontal  and  vertical  components,  is  often  desired  to  be  large  to 
promote  a  sense  of  immersiveness  (i.e.,  presence)  and  to  improve  task  performance  through  the  utilization  of 
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peripheral  vision  (limited  DFOV  displays  result  in  the  development  of  altered  scanning  strategies).  However, 
a  trade  off  that  occurs  when  one  tries  to  increase  DFOV  using  an  existing  display  is  a  corresponding  reduction 
of  screen  resolution.  Given  a  fixed  display  size,  the  only  way  to  increase  DFOV  is  to  either  magnify  the 
display  using  optics  or  move  the  eye  closer  to  the  screen.  In  either  case,  pixel  size  increases  in  the  same 
proportion  as  screen  size  (since  both  are  fixed  values).  As  pixel  sizes  increase,  display  resolution  decreases. 
Research  is  needed  to  better  understand  DFOV  requirements  for  various  UMV  applications.  Additionally, 
the  relation  between  DFOV  and  geometric  field-of-view  (zoomed  in  or  zoomed  out  images)  and  its  effect  on 
UMV  operator  performance  and  comfort  is  needed  [35]. 

6. 5. 1.3. 5  Simulator  Sickness/Cyber  Sickness 

Simulator  sickness  (also  termed  cyber  sickness)  is  a  form  of  motion  sickness  that  occurs  as  a  result  of 
experiencing  computer-simulated  visual  environments  [35].  Symptoms  include  nausea,  fatigue,  headache, 
eye-strain,  dizziness,  malaise,  and  blurred  vision.  Besides  the  deleterious  effects  associated  with  simulator 
sickness,  experiencing  these  symptoms  may  result  in  reduced  desire  to  interact  with  the  provoking  system  in 
the  future,  thus  potentially  hampering  overall  mission  effectiveness.  HMD  usage  has  been  strongly  linked 
with  increased  levels  of  simulator  sickness  in  many  studies  including  those  involving  UMVs  [17,19,20]. 
Although  some  guidelines  exist,  more  research  is  needed  to  fully  characterize  and  alleviate  simulator  sickness 
in  UMV-related  HMD  applications. 

6. 5. 1.3. 6  Other  Issues 

Other  research  issues  include  the  need  to  better  understand  and  mitigate  workload  associated  with  HMD 
usage.  Due  to  ergonomic  concerns  as  well  as  the  need  to  constantly  move  one’s  head  to  change  one’s 
viewpoint,  workload  and  fatigue  are  real  concerns  associated  with  this  technology  [19]  and  mitigation 
techniques  must  be  explored  [18].  Head  tracking  technology  and  research  is  also  needed  to  define  minimum 
accuracy  requirements  for  various  UMV  systems  and  to  enable  unencumbered  operations  [36].  Display 
brightness  and  contrast  are  also  issues,  especially  for  applications  in  outdoor  environments. 
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6.5.2  Augmented/Mixed  Reality  Technology 

One  reason  that  it  is  difficult  to  operate  a  remote  UMV  or  its  sensor  is  the  “limited  angular  view  associated 
with  many  remote  vision  platforms  [which]  creates  a  sense  of  trying  to  understand  the  environment  through 
what  remote  observers  often  call  a  ‘soda  straw’”  [1].  Another  reason  for  the  difficulty  is  the  lack  of  the 
proprioceptive  cues  that  would  normally  be  available  to  a  driver  or  pilot  situated  in  a  vehicle.  One  justification 
for  the  use  of  a  head-coupled  HMD  as  discussed  in  the  previous  section  is  to  give  the  UMV/sensor  operator  a 
proprioceptive  sense  of  the  environment:  the  operator’s  proprioceptive  sense  of  the  relative  pose  of  his  or  her 
head  is  used  as  a  surrogate  for  the  relative  pose  of  a  moving  camera,  albeit  with  mixed  results. 

An  alternative  approach  is  to  represent  multiple  sources  of  sensor  data  in  a  mixed  reality  interface. 
Conventional  interfaces  do  not  integrate  multiple  sources  of  information  into  a  coherent  representation. 
For  example,  Figure  6-6  presents  compass  direction  (upper  left),  laser  range  data  (second  from  upper  left), 
video  (upper  center),  sonar  data  (upper  right),  map  (bottom),  and  team  information  (right).  All  information 
required  for  robot  control  is  available  in  the  interface,  but  it  is  up  to  the  operator  to  integrate  this  information 
into  a  meaningful  and  useful  “picture”  of  what  is  going  on.  For  operators,  this  means  that  the  cognitive 
workload  associated  with  just  getting  enough  awareness  to  control  locomotion  can  be  high  enough  that  it  is 
difficult  to  interpret  imagery,  plan  strategies,  and  do  other  higher-level  tasks  that  are  relevant  for  the  mission. 
This  problem  is  frequently  solved  by  adding  more  humans  to  the  team  who  are  responsible  for  mission-level 
issues  [2]. 
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Figure  6-6:  A  Typical  Interface  Presents  All  Sensor  Readings  Side  by  Side. 


6.5.2. 1  Description  of  Technology 

Another  approach  to  this  problem  is  to  represent  real  sensor  information  in  an  ecological  way  [3]. 
In  the  robotic  domain,  one  important  and  useful  ecological  technique  is  the  use  of  mixed  reality  interfaces. 
Such  interfaces  combine  real  data  with  virtual  elements,  and  range  from  augmented  reality  to  augmented 
virtuality  interfaces  [4].  An  example  of  such  interfaces,  taken  from  [5],  is  shown  in  Figure  6-7.  In  this 
interface,  obstacles  detected  with  a  range  sensor  are  represented  by  barrels  in  a  virtual  world.  This  virtual 
world  is  augmented  with  video  from  the  robot.  Fusing  video,  obstacles,  and  a  representation  of  the  robot 
allow  the  operator  to  perceive  the  relationship  between  the  robot’s  “shoulders”  and  the  obstacles  in  the 
world.  This  type  of  mixed  reality  display  uses  visualization  techniques,  such  as  those  developed  in  gaming, 
as  surrogates  for  the  proprioceptive  cues  that  are  missing  in  remote  operation. 


Videoimage 
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Figure  6-7:  Integrated  Display  of  Video,  Range  Readings,  and  Robot  Representation. 
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There  are  a  number  of  different  ways  for  information  to  be  presented  in  a  mixed  reality  display.  This  section 
describes  two  techniques  and  shows  examples  from  both  UGV  and  UAV  applications.  The  two  techniques 
that  will  be  shown  are  the  chase  perspective  and  the  map-based  perspective.  Each  technique  is  appropriate  for 
certain  types  of  tasks  and  modes  of  interaction  between  the  human  and  a  UMV.  The  idea  that  different 
visualization  techniques  have  uses  in  different  situations  is  a  well-known  result  in  display  design,  in  general, 
and  in  augmented  reality-based  displays,  in  particular  [6]. 

6.5.2. 1. 1  The  Chase  Perspective 

The  chase  perspective  is  illustrated  in  Figure  6-7 .  This  perspective  presents  sensor  information  in  a  way  that 
supports  locomotion,  and  is  a  typical  representation  used  in  racing  games  because  it  allows  the  direct 
perception  of  the  relationship  between  the  vehicle  and  the  afforded  directions  of  vehicle  movement.  Similar  to 
the  goal  of  using  a  head-coupled  HMD  to  help  an  operator  understand  the  pose  of  a  movable  camera, 
the  chase  perspective  can  be  augmented  with  a  visual  representation  of  the  pose  of  the  camera  relative  to  the 
robot.  This  is  illustrated  in  Figure  6-8.  Note,  however,  that  large  panning  motions  may  require  a  shift  from  the 
chase  perspective  as  illustrated  in  Figure  6-9. 


Figure  6-8:  Representing  the  Pose  of  a  Panning  Camera. 


Figure  6-9:  Depicting  Camera  Pose  May  Require  a  Perspective  Change. 
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Similar  visualization  techniques  can  be  used  to  represent  other  information  such  as  deviations  in  terrain, 
the  focal  length  of  a  zooming  camera  [7],  and  delays  in  receiving  imagery  from  the  robot  via  quickening  [5]. 

A  chase  perspective  can  similarly  be  used  to  support  aviation  with  UAVs.  An  example  of  the  chase 
perspective  is  shown  in  Figure  6-10  [8,9].  In  this  display,  a  virtual  UAV  is  included  in  the  display  to  represent 
the  pose  of  the  UAV  relative  to  the  ground.  This  virtual  UAV  is  overlaid  on  the  video  image  received  from  the 
UAV  and  allows  the  operator  to  directly  perceive  the  attitude  of  the  aircraft  with  respect  to  the  ground. 
(In  the  figure,  two  virtual  UAVs  are  shown;  one  indicates  the  actual  pose  of  the  UAV  as  received 
from  telemetry  and  the  other  indicates  the  commanded  pose  of  the  UAV.)  The  chase  perspective  shown  in 
Figure  6-10  is  taken  from  an  interface  that  runs  on  a  5  inch  or  smaller  display. 


Figure  6-10:  The  Chase  Perspective  for  a  UAV. 


Note  that  the  chase  perspective  for  the  UAV  is  earth-centered  rather  than  pilot-centered.  When  the  operator  is 
on  the  ground,  banking  right  is  not  accompanied  by  a  pilot-perceived  change  in  the  earth’s  horizon  nor  is  it 
accompanied  by  other  vestibular  cues.  Since  the  operator  is  on  the  ground,  the  chase  perspective  adopts  a 
ground-based  perspective  wherein  a  bank  command  is  depicted  by  having  the  virtual  UAV  dip  its  wing  in  the 
commanded  direction. 

It  is  possible  to  take  this  ground-centered  perspective  a  step  further.  Since  a  fixed-mount  camera  rotates  when 
the  UAV  banks,  the  operator  must  switch  from  a  ground  perspective  to  issue  commands,  to  the  UAV 
perspective  to  interpret  video.  This  switch  might  be  a  cause  of  cognitive  workload  because  the  ground-based 
operator  must  interpret  rotations  in  the  video  caused  by  a  banking  UAV.  An  interface  that  makes  both  video 
and  bank  angle  have  a  ground-centered  reference  frame  is  shown  in  Figure  6-11.  This  interface  is  built  for  a 
control  device  called  the  PhyCon  (for  Physical  Icon)  [8].  Rather  than  using  a  handheld  computer  to  issue 
commands  to  the  UAV,  a  physical  model  of  the  UAV  is  used.  When  the  operator  banks  the  model,  the  actual 
UAV  also  banks.  Although  it  is  somewhat  difficult  to  see  in  the  figure,  the  pose  of  the  aircraft  is  projected 
onto  the  video  from  a  ground-centered  reference  frame  using  the  chase  perspective.  This  is  a  type  of  mixed 
reality  interface  [4].  (In  practice,  there  are  actually  two  virtual  depictions  of  the  UAV.  The  first  virtual  UAV  is 
depicted  using  the  actual  telemetry  from  the  UAV.  The  second  virtual  is  depicted  using  the  commanded  pose 
from  the  PhyCon.  Having  both  of  these  projected  into  the  augmented  reality  display  allows  the  operator  to  see 
that  the  actual  UAV  is  responding  appropriately  to  the  commanded  pose.) 
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Figure  6-11:  Rotating  the  Video  Supports  the  Ground-Based  Perspective  of  the  Remote  Operator. 

The  video  feed  is  digitized  and  displayed  on  a  computer  (in  this  case,  a  laptop  which  presents  the  video 
through  an  eyeglass-mounted  display).  Prior  to  presenting  the  imagery,  the  telemetry  from  the  UAV  is  used  to 
rotate  the  image  so  that  the  horizon  stays  approximately  level.  This  is  depicted  in  Figure  6-11.  Rotating  the 
image  so  that  the  horizon  stays  level  means  that  both  the  video  and  the  UAV  attitude  are  depicted  in  a  ground- 
relative  reference  frame. 

6. 5. 2. 1.2  The  Map-Based  Perspective 

The  chase  perspective  primarily  supports  locomotion.  When  it  is  necessary  to  operate  at  the  level  of 
navigation  or  planning,  it  is  often  useful  to  have  a  map  of  some  sort.  For  example,  many  potential  UMV 
missions  require  some  sort  of  exhaustive  or  heuristic  search.  Issuing  commands  for  these  searches  and 
depicting  the  progress  of  these  searches  may  be  easier  for  the  operator  if  a  map-centered  reference  frame  is 
used  [6].  The  task  is  to  present  map  information  in  a  useful  way  and  then  to  integrate  the  video  into  the  map 
using  this  map-centered  perspective. 

Figure  6-12  depicts  a  mixed  reality  display  that  integrates  a  virtual  map  with  video  from  a  robot  [10,11]. 
The  virtual  map  is  a  3D  rendering  of  a  2D  occupancy  grid  map  created  using  Konolige’s  map-building 
algorithms  and  software  [12].  The  video  is  depicted  in  this  virtual  world  in  such  a  way  that  video,  map, 
and  robot  pose  information  are  simultaneously  visible.  There  are  a  number  of  desirable  features  of  such  an 
interface,  including 

a)  The  ability  to  determine  what  has  been  searched  and  what  needs  to  be  searched; 

b)  The  ability  to  perceive  how  the  robot  is  oriented  with  respect  to  landmarks  in  the  world;  and 

c)  The  ability  to  augment  map  information  with  icons  or  other  semantic  labels. 
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Figure  6-12:  An  Occupancy  Grid  Map  Can  be  Used  as  the  Basis  for  Navigation  [10]. 

These  same  three  elements,  video,  map,  and  robot  (UAV)  pose,  can  also  be  used  to  create  a  map-centered 
display  for  UAVs.  In  this  case,  the  virtual  world  can  be  built  on  GIS  terrain  data  or  satellite  imagery,  the  UAV 
pose  can  be  depicted  on  this  map  with  either  a  north-up  or  linked  orientation  [6],  and  the  video  from  the  UAV 
can  be  projected  into  the  map  with  appropriate  rotations  so  that  the  video  is  oriented  in  the  same  reference 
frame  as  the  map.  Some  evidence  suggests  that  using  a  map-based  display  helps  operators  more  rapidly 
“cover”  a  particular  region  of  interest  by  supporting  better  navigation  [13].  The  techniques  described  in  the 
next  section  on  various  perspectives  of  the  tactical  space  for  3D  visualization  can  be  simplified  and  applied  to 
the  augmented  virtuality  display.  Such  an  interface  is  illustrated  in  Figure  6-13. 


Figure  6-13:  Augmenting  a  Terrain  Map  with  Symbology  Can 
Better  Support  Navigation  and  Sensor  Management  [23]. 

It  is  important  to  note  that  map-based  interfaces  have  been  used  to  construct  augmented  reality  displays  in 
aviation.  These  displays,  which  may  be  either  heads-up  or  head-down  and  which  may  be  retrofitted  to  older 
aircraft  [14],  are  referred  to  as  synthetic  vision  displays  [15].  Several  human  factors  studies  have  been 
conducted,  many  showing  that  there  is  an  increase  in  navigation-related  situation  awareness  with  negligible 
loss  in  aviation-related  situation  awareness  [16],  presumably  because  a  greater  fleld-of-view  and  subsequent 
sense  of  realism  can  be  obtained  with  such  displays. 
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6.5.2.2  Actual  or  Potential  Application  to  UMVs 

Application  of  these  display  technologies  to  UMVs  is  an  area  of  ongoing  work.  Research  and  development  of 
mixed  reality  displays  is  being  supported  via  Idaho  National  Laboratory  under  the  Joint  Robotics  Program 
with  the  intent  of  developing  a  fieldable  augmented  virtuality  display  for  UGVs  in  the  very  near  future. 
As  part  of  achieving  appropriate  levels  of  technology  readiness,  human  factors  studies  have  been  conducted 
that  provide  strong  evidence  that  the  mixed  reality  interfaces  described  herein: 

a)  Make  it  easier  to  tele-operate  a  robot  [5]; 

b)  Make  it  easier  to  build  a  map  of  an  area  [11,13]; 

c)  Work  well  with  UGVs  that  have  autonomy  that  allows  interaction  beyond  tele-operation;  and 

d)  Make  it  easier  to  use  a  panning  camera  [10]. 

The  mixed  reality  displays  for  UAVs  described  herein  run  on  small  displays  that  may  be  appropriate  for  a 
dismounted  control  device  of  the  type  considered  in  the  Future  Combat  Systems  program.  These  interfaces 
have  been  demonstrated  on  laboratory  UAVs  that  are  equipped  with  the  same  automated  controllers  used  on 
several  class  1  military  UAVs. 

Research  at  the  Air  Force  Research  Laboratory’s  Human  Effectiveness  Directorate  is  exploring  the  value  of 
mixed  reality  display  concepts  for  UAV  operation  [17,23].  Research  to  date  has  focused  on  improving  the 
situation  awareness  and  performance  of  a  UAV  sensor  operator  for  target  search  tasks  through  collaboration 
with  Rapid  Imaging  Software,  Inc.  (see  Figure  6-13).  A  recent  study  demonstrated  a  significant  reduction  in 
search  time  required  to  find  ground  landmarks  when  virtual  marking  flags  were  enabled.  Future  studies  will 
develop  guidelines  for  system  update  rate,  symbology  and  labelling,  declutter  techniques  and  terrain 
depiction. 

Like  the  occupancy  grid-based  displays  for  UGVs,  terrain  or  image-based  displays  for  UAVs  are  hypothesized 
to  support  better  navigation,  especially  when  navigation  is  complex  or  is  performed  under  adverse  visibility 
conditions  [15].  However,  an  argument  made  in  [18]  indicates  that  the  emphasis  on  realism  in  these  displays 
may  produce  designs  that  are  attractive  but  less  effective  than  they  should  be.  The  authors  call  for  designers  to 
avoid  “naive  realism”  by  using  caricatures,  icons,  symbology  and  other  abstracted  representations  of  the  kinds 
of  information  that  an  operator  desires  [19].  Added  to  this  caution  is  the  observation  that  overuse  of 
symbology  can  create  cluttered  displays  [20]  and  that  certain  types  of  disruptions  can  be  very  difficult  to 
recover  from  even  in  the  presence  of  clear  symbolic  labels  [21].  Importantly,  if  information  is  sufficient  to 
support  precise  navigation  via  augmented  reality  (synthetic  vision)  displays,  it  may  also  be  sufficient  to 
perform  autonomous  navigation.  This  may  be  especially  important  in  areas  such  as  search,  where  it  is 
desirable  to  ensure  that  imagery  from  the  camera  efficiently  and  completely  covers  a  region  of  the  ground  [22] 
and  where  screen  size  or  team  size  is  small  enough  that  it  is  challenging  for  an  operator  to  simultaneously 
aviate,  navigate,  and  analyze  imagery.  The  complexity  of  such  navigation  is  illustrated  in  Figure  6-14. 


6-36 


RTO-TR-HFM-078 


^■nato 

WP  OTAN 


ADVANCED  UMV  OPERATOR  INTERFACES 


Figure  6-14:  UAV  Locations  (thin  blue  lines)  Required  to  Support  Low  Ground  Speed, 
Complete  Coverage  by  a  Camera  Footprint  Spiral  (bold  red  line). 


Perhaps  the  most  important  potential  application  of  mixed  reality  displays  of  the  type  described  herein  is  in 
forming  a  common  operating  concept  for  many  types  of  UMVs.  The  displays  rely  heavily  on  visualization 
techniques  used  in  graphics  and  gaming,  and  therefore  have  some  basis  in  being  useful  for  a  large  class  of 
operators  in  a  large  variety  of  situations.  Moreover,  the  mixed  reality  displays  described  for  UAVs  are  similar 
in  concept  to  synthetic  vision  systems. 


6.5.2.3  Technology  Maturity,  Challenges,  and  Unresolved  Issues 

The  example  interfaces  described  in  this  section  are  in  the  alpha  or  early  beta  stages  of  development. 
They  have  all  been  tested  on  physical  platforms  and  there  have  been  tests  that  confirm  their  usefulness  in 
limited  problems.  A  number  of  challenges  and  unresolved  issues  remain.  The  most  pressing  concern  is  the 
effects  of  terrain  on  visualization  and  control  for  both  the  UAV  and  UGV  platforms.  UGVs  must  frequently 
operate  in  outdoor,  unstructured  environments.  While  there  is  mounting  evidence  that  in  worlds  with  a  level 
ground  plane,  such  as  inside  a  building,  the  UGV  mixed  reality  interfaces  work  well,  it  is  an  open  area  how  to 
adapt  these  interfaces  to  uneven  terrain.  A  similar  challenge  exists  with  the  UAVs.  For  class  1  UAVs, 
the  operational  altitude  is  frequently  very  close  to  the  ground.  Since  the  chase  perspective  and  map 
perspective  can  make  it  difficult  to  convey  information  about  height  above  ground  to  the  operator,  is  may  be 
desirable  to  use  a  terrain  model  and  height  above  ground  sensor  to  autonomously  maintain  a  consistent  height 
above  ground. 

Another  very  important  challenge  is  to  identify  how  to  transition  from  one  display  perspective  (e.g.,  chase 
view)  to  another  perspective  (e.g.,  map-view)  when  the  operator  shifts  from  one  mission  phase  to  another. 
Questions  such  as  whether  the  display  should  automatically  adapt  to  such  shifts  or  whether  the  operator 
should  explicitly  command  display  changes  are  important  to  answer.  Fortunately,  there  is  a  considerable 
literature  on  mode  confusion,  adaptive  displays,  automation  management  policies,  and  so  on. 

Other  important  challenges  remain.  These  challenges  include  the  following: 

•  Understanding  the  effects  of  confounding  factors  such  as  screen  size,  communications  delay,  and 
sunlight  readability. 
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•  Learning  how  to  create  virtual  representations  of  shape  shifting  robots  (e.g.,  the  PackBot)  or  robots 
with  manipulators. 

•  Identifying  whether  robot  health  information  should  be  displayed  as  part  of  the  virtual  UMV 
representation,  or  as  a  separate  part  of  the  display. 

•  Learning  how  to  represent  the  intent  of  the  UMV’s  autonomy. 

•  Coordinating  the  mixed  reality  displays  with  other  display  concepts  such  as  those  required  to  support 
multiple  robots,  semantic  mark-up  of  the  map,  symbology,  and  display  declutter. 
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6.5.3  3D  Visual  Displays 

Displayed  information  should  fit  the  nature  of  the  optimal  mental  processing  operations  required  to  perform 

the  task.  Wickens  and  Andre  [1],  Haskell  and  Wickens  [2],  and  Wickens  and  Carswell  [3]  propose  an 

interaction  between  the  type  of  task  performed  and  the  type  of  display  most  suited  for  that  task. 
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UAV  supervisory  control  is  an  integrated  task  since  the  supervisor/operator  must  understand  and  combine 
location,  angular,  and  rate  of  change  information  in  3D.  Because  3D  displays  show  the  necessary  information 
within  a  single  spatial  representation,  it  is  proven  that  operator  performance  benefits  more  from  these 
integrated  3D  displays  than  from  displays  representing  this  spatial  information  in  separate  dimensions. 

In  this  respect,  we  distinguish  two  types  of  3D  displays: 

•  3D  perspective  displays.  Perspective  information  may  be  considered  as  a  subset  of  three-dimensional 
information  (i.e.,  a  stereoscopic  image  without  binocular  disparity). 

•  3D  stereoscopic  displays.  Stereoscopic  displays  are  displays  that  create  a  true  sense  of  depth. 


6.5.3. 1  Description  of  Technology 

6. 5. 3. 1.1  3D  Perspective  Displays 

The  graphical  representation  of  the  external  world  can  be  shown  from  the  position  of  the  observer,  egocentric 
display  information,  or  from  a  position  somewhere  in  space,  exocentric  display  information  [4]). 
An  egocentric  presentation  shows  the  external  world  only  in  one  direction,  suited  to  local  guidance  tasks, 
i.e.,  following  a  planned  navigational  track  with  limited  preview.  For  example,  when  considering  a  flight  task, 
pilots  perceive  themselves  flying  through  the  environment  as  seen  from  an  ego-referenced  frame  [5]). 
This  means  that  the  display  direction,  left  or  right,  always  corresponds  with  the  control  direction.  Exocentric 
displays  separate  the  observer’s  eye  point  and  actual  position,  showing  the  external  world  that  surrounds  the 
observer,  thus  assisting  with  global  awareness.  They  represent  the  world  either  in  a  fixed  geographical 
co-ordinate  system  (world-referenced;  e.g.,  north  up)  or  with  respect  to  one’s  momentary  position  and 
orientation  (ego-referenced;  e.g.,  track  up  or  heading  up). 

Research  using  egocentric  perspective  displays  mainly  has  examined  the  navigation  accuracy  during  local 
guidance  tasks  [6, 7, 8, 9].  Research  using  exocentric  perspective  displays  mainly  focussed  on  world-referenced 
aspects:  How  effective  will  these  displays  support  the  situation  awareness  in  a  geographical  environment? 
One  study  [10])  investigated  the  flight  accuracy  and  orientation  of  pilots  using  two-dimensional  (2D) 
plan-view,  and  3D  perspective  north  up,  track  up  and  heading  up  situation  displays.  Other  investigations  have 
examined  pilots’  perceptions  of  the  geographical  environment  [11]  and  the  assessment  of  collision  risk  [12]. 
Results  have  revealed  that  world-referenced  exocentric  displays  can  increase  pilots’  geographical  orientation, 
but  can  hamper  pilots’  tracking  performance  in  local  guidance  tasks  because  of  the  required  mental 
transformations.  For  example,  a  north  up  display  may  cause  confusion  in  an  aircraft  heading  south 
[2,10].  Note  that  these  investigations  used  more  or  less  static  scenarios.  Ego-referenced  exocentric  display 
information,  supporting  the  orientation  of  objects  in  space  relative  to  the  observer’s  momentary  position, 
has  hardly  been  a  topic  of  much  research  activity.  However,  one  may  suppose  that  knowledge  about  the 
position  of  surrounding  objects  in  space  is  of  major  importance. 

Other  research  [13,14]  has  addressed  methods  of  presenting  perspective  information  on  a  display, 
investigating  factors  that  influence  the  judgment  of  spatial  information:  grid-surface  density,  Geometric  Field 
Of  View  (GFOV),  Station  Point  Distance  (STP),  and  target  distance.  It  appeared  that  a  perspective  graphical 
presentation  of  the  airspace  provides  a  more  natural  (ecological)  and  compatible  representation  than  a 
conventional  plan-view  display  [15],  but  it  was  found  difficult  to  estimate  the  exact  position  of  computer¬ 
generated  objects  in  that  space.  It  is  necessary,  though,  to  carefully  select  the  design  parameters  of  the  spatial 
information.  For  example,  incorrect  combination  of  GFOV  and  STP  causes  deformation  of  the  presented 
image  which  leads  to  overestimation  of  the  elevation  angle  [16]. 
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Another  important  factor  that  affects  the  interpretation  of  perspective  display  information  is  scene  dynamics 
during  motion  -  the  relative  movement  of  the  graphical  components.  In  this  respect,  [17]  distinguished  motion 
references  in  avionic  systems:  with  inside-out  or  ego-centered  motion  reference,  the  horizon  rotates  according 
to  roll  and  pitch  whereas  the  aircraft  symbol  remains  stationary;  and  with  outside-in  or  earth-centered  motion 
reference,  the  horizon  is  stationary  whereas  the  aircraft  symbol  rotates.  Inside-out  motion  reference  is 
compatible  with  the  motion  of  the  environment  as  it  is  observed  from  the  cockpit;  the  aircraft  is  the  reference, 
and  the  world  is  rotating.  In  contrast,  outside-in  motion  reference  shows  the  movements  of  the  aircraft  in  a 
stationary  world,  representing  the  aircraft  as  a  dynamic  element  in  the  real  world.  Research  concerning  the 
graphical  representation  of  perspective  display  information  as  well  as  motion  reference  is  reported  here. 

van  Breda  and  Veltman  [18]  investigated  3D  perspective  displays  in  a  simulated  flight  task.  An  aircraft 
guidance  task  was  chosen  with  the  following  instruction:  Perform  a  target  acquisition  task  with  a  fighter 
aircraft,  i.e.,  first  locate  a  target  that  appears  and  then  perform  target  interception  (point  the  aircraft’s  nose 
toward  the  target  as  quickly  as  possible).  For  this  task,  it  is  of  vital  importance  that  pilots  have  correct 
estimations  of  the  target’s  position,  and  of  the  route  to  be  followed  toward  the  target. 

Egocentric  displays  are  less  suitable  for  this  task  because  they  provide  only  a  limited  view  of  the  airspace  in 
the  flight  direction.  For  high-quality  task  performance  it  is  essential  that  the  pilot  obtain  full  preview,  that  is, 
being  able  to  fully  perceive  the  course  of  the  stimulus  (forcing  function).  Exocentric  displays  may  meet  this 
requirement,  because  targets  beside  or  even  behind  are  shown  [19].  Both  the  display  types  were  used  in  an 
experiment:  an  egocentric  heading-up  display  for  the  initial  aircraft  following  task  (Figure  6-15)  and  an 
exocentric  radar  display  for  the  interception  task  (Figure  6-16).  Of  the  latter  display,  two  types  were 
investigated:  a  plan-view  2-D  radar  display,  and  a  perspective  3D  radar  display.  In  the  2D  display  type,  radar 
information  consisted  of  an  augmented  circular  plan-view  display.  The  display  centre  represented  the  pilot’s 
aircraft;  a  target  symbol  indicated  the  target  position  relative  to  the  pilot’s  aircraft.  The  display  was  augmented 
with  colour  coding  for  relative  target  position,  and  a  separate  scale  for  target  elevation  was  used.  In  the 
3D  condition,  radar  information  consisted  of  an  exocentric  perspective  spherical  display,  depicting  the 
surrounding  airspace  as  a  dot  pattern.  Pilot’s  performance  was  analysed  and  evaluated  in  terms  of  target 
acquisition  time,  tracking  performance  and  mental  workload. 


Figure  6-15:  The  Main  Display. 
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Figure  6-16:  Overview  of  the  Investigated  Display  Types. 


The  earth  is  represented  by  a  grid,  the  target  by  an  Xsymbol,  the  flight  direction  by  a  +  symbol,  and  the  length 
axis  of  the  intercepting-aircraft  by  an  _  symbol.  Additional  indicators  for  airspeed,  vertical  speed,  altitude, 
and  heading  are  shown.  The  radar  display  is  presented  in  the  lower  left,  in  this  case  a  two-dimensional  plan- 
view  radar  image  with  an  additional  target  elevation  indicator. 

The  left  figure  shows  a  2-D  radar  display  with  a  separate  target  elevation  indicator;  the  right  figure  shows  a 
3D  radar  display  with  a  separate  target  distance  indicator.  The  intercepting-aircraft  symbol  is  always  presented 
in  the  display  centre.  The  line  ahead,  perpendicular  to  the  wing  plane,  is  the  visor.  Two  3D  configurations 
were  investigated:  outside-in  motion  reference  (the  sphere  with  horizon  and  target  symbol  remain  horizontal, 
whereas  the  intercepting-aircraft  symbol  rotates  as  a  function  of  pitch  and  roll)  and  inside-out  motion 
reference  (the  intercepting-aircraft  symbol  with  wing  plane  remains  horizontal,  whereas  the  sphere  with 
horizon  and  target  symbol  rotates  as  a  function  of  pitch  and  roll).  This  figure  shows  a  3D  radar  display  with 
outside-in  motion  reference. 

The  experimental  results  showed  strong  benefits  of  perspective  displays  for  situation  awareness  support  in  the 
target  acquisition  task.  A  considerable  reduction  in  the  target  acquisition  time  was  obtained  when  pilots  used  a 
perspective  radar  display  instead  of  a  conventional  plan-view  display  in  the  cockpit  (Figure  6-17). 
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display  type 


Figure  6-17:  Target  Acquisition  Time  as  a  Function  of  Two  Dimensional  (2-D)  and  Three  Dimensional 
(3D)  Display  Type  and  Initial  Position  of  the  Target  Aircraft,  Averaged  Across  Participants. 


This  finding  confirms  one  of  the  most  important  benefits  of  3D  perspective  display  representation  as  was 
observed  by  Wickens  et  al.  [5]:  The  elimination  of  the  time-consuming  scanning  that  is  necessary  to  go  back 
and  forth  between  the  several  parts  of  a  display.  In  the  2-D  display,  a  circular  radar  image  indicating  target 
azimuth  and  a  separate  linear  indicator  for  the  target  elevation  had  to  be  scanned  and  mentally  combined  for 
target  position  estimation.  In  both  the  3D  perspective  displays,  target  azimuth  and  elevation  were  presented  in 
a  single  object.  In  the  3D  perspective  sphere,  both  the  target  elevation  and  target  azimuth  were  integrated. 
For  target  acquisition,  these  variables  represent  information  of  close  mental  proximity. 

Inside-out  motion  reference  provided  a  direct  relationship  between  the  displayed  movement  of  the  scene  and 
the  perceived  movements  of  surrounding  objects.  The  display  elements  representing  the  outside  world  - 
three  in  this  case  -  the  sphere  (globe)  of  the  perspective  radar  display,  the  displayed  horizon  in  the  main 
(local  guidance)  display,  and  the  visual  horizon  as  seen  from  the  cockpit,  were  consistent  in  this  display, 
making  the  presented  information  ecologically  valid  [15,20].  The  tracking  task  could  therefore  be  considered 
as  a  natural  process.  The  perspective  sphere  was  presented  by  a  dot  pattern,  providing  adequate  preview  for 
tracking  [21].  As  was  observed  by  [4]),  3D  displays  can  be  used  very  efficiently  for  local  guidance  tasks: 
in  the  current  experiment  the  target  acquisition  time  was  reduced  by  more  than  40%.  It  is  obvious  that  this  is  a 
major  improvement  in  performance. 

The  subjective  effort  scores  showed  almost  the  same  pattern  as  the  performance  data.  The  pilots  felt  that  less 
effort  was  needed  when  perspective  displays  were  used;  in  particular,  they  felt  more  comfortable  with  the 
inside-out  motion  display. 

The  strongest  motive  to  ‘go  three-dimensional’  is  the  inability  to  combine  the  two  dimensional  ‘bird’s-eye 
view’  with  a  graphical  presentation  of  altitude  and  depth  information.  As  a  consequence,  in  all  current  systems 
altitude  and  depth  information  is  presented  as  numerical  read-outs.  Representations  lacking  integrated  altitude 
and  attitude  information  complicate  situation  assessment  in  two  ways  [22].  Data  that  are  difficult  to  acquire 
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are  more  difficult  to  use  in  making  a  decision.  Also,  without  immediately  evident  altitude  information, 
a  decision  maker  may  substitute  arbitrary  or  situation-biased  altitudes,  that  may  be  difficult  to  supplant  even 
when  the  actual  data  are  presented.  With  more  realistic  images  of  the  environment  and  tracks  in  a 
3D  perspective  (see  Figure  6-18),  decision  makers  argue,  the  interface  becomes  more  natural  and  less  effort  is 
required  to  comprehend  the  current  situation.  It  eliminates  the  burden  of  integrating  and  interpreting  of 
multiple  representations,  abstract  symbols,  and  textual  read-outs.  Some  earlier  experimental  results  with 
perspective  displays  confirm  these  expectations. 


Figure  6-18:  Example  of  a  3D  Perspective  Display  (Adapted  from  Denehy,  Johns  Hopkins  APL). 

With  the  potential  to  improve  performance,  3D  perspective  or  stereoscopic  displays,  however,  can  also  have 
their  drawbacks.  Inherent  to  the  perspective  view,  objects  are  presented  larger  or  smaller  as  a  function  of  the 
operators  viewing  distance,  location  and  angle.  Objects  close  to  the  operator  will  be  shown  with  much  more 
resolution  than  objects  at  larger  viewing  distances.  In  many  cases  these  differences  will  not  necessarily  reflect 
differences  in  tactical  relevance  and  meaning.  In  their  aspiration  to  design  more  realistic  or  natural 
representations  of  the  environment  researchers  and  software  engineers  also  prefer  to  apply  the  principle  of 
immersion  to  create  the  feeling  of  being  part  of  that  environment.  Becoming  embedded  in  the  situation, 
however,  can  have  some  serious  disadvantages.  What  you  see  becomes  dependent  on  your  own  orientation. 
You  have  to  look  around  not  only  to  see  what  is  happening  miles  away  in  front  of  you,  but  also  what’s 
happening  directly  behind  your  back.  More  realistic  does  not  necessarily  mean  more  functional. 

One  approach  used  at  TNO  is  to  let  the  environment  become  a  3D  object  in  itself,  such  as  a  transparent  cube 
that  can  be  viewed  from  the  outside  and  easily  manipulated  to  see  the  environment  from  different  perspectives 
and  in  different  scales.  In  Figure  6-19,  two  pictures  are  shown  that  give  an  impression  of  this  concept  of  a 
3D  tactical  space.  Simple  geometric  2D  and  3D  symbols,  comparable  to  the  familiar  naval  tactical  display 
symbology,  are  used  to  represent  tracks. 
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Figure  6-19:  Examples  of  the  Different  Perspectives  from  which  the  Tactical  Space  Can  be  Viewed. 


With  one’s  own  ship  in  the  center  of  the  tactical  space  as  default,  the  operator  has  independent  controls  for 
horizontal  and  vertical  zoom-in  and  zoom-out,  to  change  the  range  or  ceiling  for  which  contacts  are  displayed. 
Size  of  symbols  and  track  labels  is  held  constant,  irrespective  of  the  zooming  factor.  With  an  off-center 
function,  other  tracks  than  the  own  ship  can  be  selected  to  be  presented  in  the  center  of  the  tactical  space. 
By  turning  around  and  tilting  the  cube,  the  operator  can  view  the  environment  from  almost  every  perspective, 
be  it  air,  surface  or  even  subsurface,  underwater.  Independent  of  the  selected  perspective,  all  symbols  and 
track  labels  remain  in  a  steady  orientation  towards  the  user  to  guarantee  good  legibility.  With  a  time-offset 
function  the  user  can  both  playback  or  consult  preceding  moments  in  the  situation  and  extrapolate  the 
situation  towards  a  future  point  in  time.  And  dependent  of  platform  type  and  armament,  critical  points  in 
sensor  and  weapon  coverage  can  be  shown. 

One  of  the  most  important  functions  of  the  tactical  space  is  to  have  an  environment  that  integrates  information 
from  different  sources  such  as  intel,  traffic  management,  sensor  and  geographic  information  in  a  tactically 
meaningful  way  to  support  full  situation  awareness  and  assessment,  especially  when  situations  become  less 
predictable  or  more  complex  as,  for  example,  along  the  seashore.  The  concept  of  the  tactical  space,  however, 
does  not  mean  that  all  other  ways  of  information  display  become  superfluous,  as  confirmed  in  research  done 
for  the  U.S.  Navy  [23].  There  is  no  single  display  that  can  offer  all  information  needed  in  an  optimal  way. 
The  secret  to  good  information  presentation  often  lies  in  the  diversity  of  multiple  views  on  a  situation 
and  different  graphical  formats.  With  the  tactical  space  well  suited  for  the  higher  level  tactical  assessment, 
the  2D  bird’s  eye  view  for  instance,  with  its  fixed  orientation,  more  readily  suits  the  fast  localization  of  a 
track. 

6. 5. 3. 1.2  3D  Stereoscopic  Displays 

Simply  stated,  3D  technology  adds  the  sense  of  depth  by  imitating  one  or  more  of  the  visual  depth  cues. 
Here  we  describe  how  the  3D  technologies  achieve  this  result.  A  good  overview  can  be  found  at  the  following 
website:  http://www.stereo3d.com/3dhome.htm.  More  detail  on  the  human  factors  aspects  can  be  found 
elsewhere  [24]. 
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6.5.3 . 1 .2. 1  Convergence 

Convergence  can  be  activated  by  presenting  (slightly)  different  images  to  the  left  and  right  eyes.  The  left-right 
difference  is  called  the  stereoscopic  disparity.  The  most  common  methods  are  shutter  glasses,  polarized 
glasses,  Red/Green  glasses,  and  Head-mounted  displays.  They  share  the  common  disadvantage  of  constraining 
the  user.  Most  importantly,  eye  contact  is  disturbed,  hampering  communication  with  others.  These  devices  are 
therefore  not  well  suited  for  group  activities. 

To  avoid  the  constraints  imposed  by  wearing  optics  in  front  of  the  eyes,  so-called  “auto-stereoscopic 
3D  displays”  are  being  developed.  The  word  “auto”  signifies  that  the  user  does  not  need  to  wear  an  optical 
device.  The  optics  are  incorporated  in  the  display,  splitting  the  image  in  a  left  eye  and  a  right  eye  component 
at  the  display  instead.  These  optics  are  typically  called  “lenticular  screens”,  and  are  glued  to  the  flat-panel 
display.  A  lenticular  screen  consists  of  small  lenses  that  bend  the  light  from  different  display  pixels  in 
different  directions.  An  inherent  feature  of  auto-stereoscopic  displays  is  that  the  head  needs  to  be  positioned  at 
the  right  place.  If  for  example  the  right  eye  is  shifted  6  cm  to  the  left,  it  will  see  the  left  eye  image.  Though 
solutions  exist  that  allow  some  freedom  of  head  movement,  a  price  is  paid  in  terms  of  an  increase  in  cross-talk 
which  reduces  the  viewing  comfort  [25]  or  in  terms  of  added  complexity  in  the  form  of  a  head  tracker  [26]. 
For  a  comparison  of  3D  methodologies  on  visual  comfort  see  [27].  The  disadvantages  of  the  four  main 
techniques  can  be  summarized  as  follows: 

•  Shutter  glasses:  low  luminance,  flicker  in  daylight  environments 

•  Polarised  glasses:  need  to  keep  the  head  straight  up 

•  Red/Green  glasses:  no  color  vision,  chromatic  aberration,  cross-talk 

•  HMDs:  image  moves  with  the  head,  cables  or  weight 

In  the  next  section,  a  fifth  technique  (transparency)  will  be  described. 

6. 5. 3. 1.2. 2  Pictorial  Depth  Cues 

Displays  that  contain  symbols  are  oftentimes  incompatible  with  the  use  of  monocular  depth  cues  because 
these  cues  tend  to  interfere  with  the  clarity  and  standardization  of  the  symbols. 

6.5.3. 1 .2.3  Accommodation  and  Parallax 

The  3D  displays  described  above  simulate  the  convergence  depth  cue,  but  do  not  provide  accommodation  and 
parallax,  which  means  that  the  depth  percept  is  incomplete.  Parallax  can  be  added  by  tracking  the  head 
movements  and  adjusting  the  view  point  accordingly.  However,  even  with  a  fairly  powerful  computer  a  time 
delay  between  head  movement  and  image  adjustment  remains  noticeable.  Except  for  one  prototype  3D  display 
in  Oxford  [28],  the  accommodation  cue  can  only  be  added  by  imaging  the  scene  at  physically  different 
distances.  The  most  advanced  system  is  the  U.S.  Navy  sponsored  “volumetric  display”  which  achieves  the 
accommodation  cue  by  imaging  on  a  rotating  drum  [29].  However,  its  large  volume  (approximately  1  m3) 
makes  it  unsuitable  for  the  type  of  applications  we  have  in  mind. 

A  relatively  simple  way  to  include  accommodation  and  parallax  in  the  depth  percept  is  to  optically 
superimpose  two  or  more  image  slices  through  the  scene.  Such  a  transparent  display  presents  “true  depth” 
in  the  sense  that  all  depth  cues  are  present  except  for  occlusion.  An  example  transparent  display  is  shown  in 
Figure  6-20.  A  New  Zealand  Company  [30]  is  the  first  to  have  marketed  a  compact  transparent,  2-plane 
display.  The  display  consists  of  two  LCD  filters,  one  placed  in  front  of  the  other,  making  a  subtractive 
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transparent  display.  Recently  a  20-plane  transparent  display  [31]  has  come  on  the  market.  We  believe  these 
developments  to  be  highly  significant,  as  argued  in  the  next  section. 


Figure  6-20:  The  Experimental  Transparent  3D  Setup  at  TNO  Human  Factors. 

In  Figure  6-20,  two  images  are  combined  with  a  half-silvered  mirror.  Because  the  light  from  the  two  displays 
adds  up,  we  call  it  an  additive  transparent  display.  The  experiments  examine  the  influence  of  various  design 
parameters  like  amount  of  depth  and  scene  content. 

6. 5. 3. 1.3  Transparent  Depth  Displays 

6.5.3. 1 .3. 1  Limited  Number  of  Depth  Planes 

So  far  not  much  research  has  been  done  on  transparent  displays.  This  is  probably  because  the  limited  number 
of  depth  planes  makes  them  unsuitable  for  the  display  of  3D  pictures  and  videos.  The  technologies  described 
above  can  in  principle  present  as  many  depth  planes  as  the  number  of  pixels  in  the  display.  We  believe 
however  that  for  professional  applications  involving  the  display  of  symbols,  2  or  3  depth  planes  will  be 
sufficient  to  provide  a  large  operational  advantage.  By  analogy,  many  of  the  “full  colour”  cockpit  displays  by 
no  means  fully  exploit  their  colour  gamut;  often  a  display  only  contains  the  four  primary  colours.  Similarly 
the  information  content  of  control  displays  often  times  can  be  naturally  divided  into  two  (friend  and  foe) 
or  three  (above,  below,  and  on  the  surface)  layers.  We  therefore  argue  that  the  advantages  of  a  transparent 
depth  display  will  out- weigh  the  disadvantage  of  the  limited  number  of  depth  planes. 

6.5.3. 1 .3.2  Optimal  Viewing  Comfort  and  Depth  Perception 

In  the  case  of  transparent  depth  displays  the  depth  percept  is  truly  extra.  The  user  does  not  pay  a  price  in  terms 
of  resolution,  colour,  viewing  angle,  the  need  for  special  glasses,  luminance,  or  viewing  comfort  as  is  the  case 
with  all  other  3D  displays.  Secondly,  our  present  research  shows  that  the  depth  percept  “pops-out” 
immediately  while  the  other  types  of  3D  displays  require  some  amount  of  time  for  the  depth  percept  to  build 
up.  Figure  6-21  shows  how  large  the  perceptual  time  delay  is  for  unfavorable  3D  stimuli.  Thirdly,  thanks  to 
the  parallax,  occlusion  of  one  object  by  another  can  easily  be  eliminated  by  moving  the  head  sideways  or 
vertically.  This  is  important  if  two  objects  are  located  at  the  same  x,  y  coordinates  but  different  heights. 
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We  therefore  believe  transparent  planes  to  be  very  promising  for  the  representation  of  computer  generated 
data  and  symbols. 


Figure  6-21:  Data  Substantiating  the  Claim  that  Accommodation  (A)  and  Motion  Parallax  (P) 
Substantially  Influences  the  Ease  of  Depth  Perception  when  the  Depth  Gradient  is  Large. 


Figure  6-21  shows  the  extra  time  required  to  perceive  the  depth  relationship  of  two  adjacent  dots  when  a 
distracting  object  is  added  at  a  different  depth.  The  horizontal  axis  contains  the  amount  of  depth  difference. 
The  increase  in  reaction  time  caused  by  the  distractor  is  1  to  2  seconds  greater  for  the  common  type  of 
3D  displays  (C:  Convergence  cue  only)  than  for  transparent  depth  displays  (CAP:  accommodation  and 
parallax  as  well  as  the  convergence  depth  cue).  These  results  imply  that  transparent  depth  displays  are  more 
natural  to  view  than  the  standard  3D  displays  described  in  Chapter  3  and  particularly  suited  for  cluttered 
3D  imagery. 

6.5 .3 . 1 .3 .3  Subtractive  and  Additive  Transparency 

Another  topic  of  current  research  at  TNO  Human  Factors  is  design  of  the  image  content.  We  have  shown  that 
a  transparent  depth  display,  if  designed  wrong,  can  be  very  hard  to  fuse  [32].  Secondly,  the  content  of  an 
additive  transparent  display  (the  front  plane  adds  light:  Figure  6-21)  needs  to  be  designed  differently  than  a 
subtractive  transparent  display  (the  front  plane  removes  light:  e.g.,  Deepvideo  [30]).  The  back  layer  of  an 
additive  display  needs  to  be  mostly  dark,  of  a  subtractive  display  mostly  white.  Otherwise  the  information  in 
the  other  layer  will  not  show  up. 

6. 5. 3. 1.3.4  Occlusion 

The  3D  displays  listed  above  do  fully  show  occlusion,  the  phenomenon  that  the  front  object  ‘hides’  the  back 
object.  An  additive  3D  display  however  is  not  able  to  show  occlusion  and  a  subtractive  3D  display  only 
partially.  In  addition,  colour  mixing  leads  to  erroneous  mixed  colours:  the  colour  of  the  front  object  is 
influenced  by  the  object  behind  and  vice  versa.  For  example,  a  yellow  object  on  a  purple  background  will 
appear  as  red  on  a  subtractive  3D  display  (Figure  6-22).  For  any  application  involving  warning  symbols  this  is 
unacceptable  and  is  at  the  expense  of  unwanted  colour  mixing.  We  therefore  argue  that  only  an  additive  + 
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subtractive  transparent  3D  display  will  support  all  the  depth  cues  listed  in  Section  6. 5. 3. 1.2.  This  may 
therefore  well  be  the  next  major  step  forward  in  3D  technology. 


Figure  6-22:  The  Effect  of  Additive  (left)  and  Subtractive  (middle)  Transparency: 
the  Colours  Combine,  Easily  Causing  Confusion. 

In  this  example  yellow  and  purple  combine  to  orange  and  red  respectively.  The  picture  to  the  RIGHT  shows 
occlusion,  the  yellow  triangles  positioned  in  front  occluding  the  purple  triangles. 


6.5.3.2  Actual  or  Potential  Application  to  UMVs 

De  Vries  and  Jansen  [33]  developed  a  simulation  environment  in  which  human  factors  principles  for  UAV 
camera  control  are  demonstrated  and  in  which  experimental  studies  are  conducted.  In  an  experiment  they  used 
this  simulator  environment  to  investigate  the  benefits  of  a  3D  map  with  regard  to  operator  performance  and 
mental  workload  (Figure  6-23).  They  constructed  a  2D  map  (oriented  north-up)  in  which  feedback  about  the 
control  input  was  provided  by  means  of  an  additional  footprint  that  showed  the  predicted  viewpoint  of  the 
camera.  Operators  were  requested  to  find  targets  on  roads  and  along  wood  edges.  They  could  use  the  map  to 
see  to  which  part  of  the  environment  the  camera  was  oriented.  In  one  half  of  the  conditions  a  3D  map  was 
available  together  with  the  2D  map.  The  3D  map  showed  identical  information,  but  was  presented  from  the 
viewpoint  of  the  camera.  In  some  conditions,  the  quality  of  the  camera  images  was  manipulated  by 
introducing  a  time  delay  of  1  second,  or  by  lowering  the  update  rate  of  the  camera  images  to  3  Hz.  This  had 
no  effect  on  the  2D  and  3D  map.  Furthermore,  in  one  half  of  the  conditions  a  secondary  task  had  to  be 
performed.  This  was  done  to  see  whether  operators  had  more  spare  mental  capacity  in  case  the  3D  map  was 
available.  Several  performance  measures  were  taken.  Workload  was  assessed  using  subjective  and 
physiological  measures. 
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Figure  6-23:  The  Left  Panel  Shows  the  2D  Map  with  the  Position  of  the  UAV  in  the  Center. 

The  right  panel  shows  the  3D  map,  which  is  drawn  from  the  viewpoint  of  the  camera.  In  both  displays, 
the  yellow  footprint  shows  the  part  of  the  environment  that  corresponds  to  the  camera  image  and  the  orange 
footprint  shows  the  predicted  position  based  on  operator  input  signals.  The  camera  image  is  presented  at  the 
bottom  of  the  right  display. 

The  left  panel  of  the  simulator  displayed  a  detailed  2D  map  of  the  environment,  north-up  oriented.  Apart  from 
terrain  information  (roads,  woods,  buildings,  etc.)  the  following  relevant  information  was  presented: 
waypoints  and  the  route  of  the  UAV  were  shown  (yellow  line),  flight  direction  of  the  UAV  (orange  arrow) 
and  the  actual  and  predicted  footprints  (see  ‘Footprint’  below  for  an  explanation).  The  right  panel  of  the 
simulator  displayed  the  3D  map,  a  virtual  3D  world  presented  from  the  viewpoint  of  the  UAV  camera. 
The  viewpoint  for  generating  the  3D  map  depended  on  the  camera  control  input  of  the  operator,  and  the 
angular  motion  of  this  viewpoint  was  identical  to  the  angular  motion  of  the  camera. 

Both  the  2D  and  3D  map  showed  a  yellow  footprint,  representing  the  section  of  the  map  that  corresponded  to 
the  camera  images.  The  orange  footprint  provided  direct  feedback  about  the  control  input.  In  the  conditions 
with  low  update  rates  and  time  delays,  the  yellow  footprint  followed  the  orange  footprint.  The  size  of  the 
footprint  could  be  adjusted  by  the  zoom  function,  providing  feedback  about  the  zoom  setting.  With  a  zoom 
factor  of  1,  the  size  of  the  footprint  on  the  right  display  was  identical  to  the  size  of  the  camera  panel. 
In  conditions  in  which  the  3D  maps  were  not  drawn  on  the  right  display,  the  footprints  remained  visible  to 
provide  feedback  about  the  zoom  settings. 

The  most  important  results  of  the  experiment  are  presented  here.  The  3D  map  significantly  improved  task 
performance  (Figure  6-24).  A  larger  area  was  inspected,  performance  on  secondary  task  improved,  indicating 
that  the  participants  had  more  spare  mental  capacity,  and  the  participants  reported  lower  effort. 
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Figure  6-24:  Percentage  of  Inspected  Areas  as  a  Function 
of  3D  Map  and  the  Quality  of  Camera  Images. 

The  positive  effect  of  the  3D  map  was  largest  when  the  quality  of  the  camera  images  was  low.  Note  that 
adequate  feedback  about  time  delays  and  low  update  rates  was  always  available  in  both  the  2D  and  3D  map 
display.  Without  this  information,  performance  would  be  much  worse  in  the  conditions  with  low  update  rates 
and  long  time  delay. 

The  subjective  effort  measure  showed  substantial  effects  as  a  function  of  all  experimental  factors 
(Figure  6-25),  however,  only  heart  rate  showed  a  small  effect  as  a  function  of  the  secondary  task.  This  may  be 
due  to  the  lower  sensitivity  of  physiological  measures  for  mental  effort.  We  found  such  discrepancies  between 
subjective  (Rating  scale  Subjective  Mental  Effort,  RSME;  [34])  and  physiological  measures  more  often  [35] 
depending  on  the  type  of  task  that  is  evaluated.  Participants  most  often  give  higher  effort  ratings  when  a  task 
becomes  more  difficult  as  a  result  of  a  reduced  quality  of  information. 
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Figure  6-25:  Subjective  Effort  Rating  as  a  Function  of  Quality  of  the  Camera  Images  and  3D  Map. 
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Physiological  measures  most  often  do  not  show  differences  in  these  situations,  because  investing  more  effort 
most  often  will  not  result  in  better  task  performance.  When  an  additional  task  has  to  be  performed,  attention 
has  to  be  divided  between  more  tasks.  In  these  situations,  additional  effort  has  to  be  invested  in  order  to  keep 
an  adequate  level  of  performance  of  the  main  task.  This  is  reflected  in  both  subjective  and  physiological 
measures.  The  difference  between  the  subjective  and  physiological  measures  can  be  explained  along  this  line 
of  thought.  Degrading  the  quality  of  the  display  makes  the  task  more  difficult,  resulting  in  higher  effort 
ratings,  but  does  not  affect  physiological  measures  because  investing  more  effort  will  not  improve 
performance.  For  the  secondary  task,  more  effort  was  required  in  order  to  maintain  an  adequate  level  of 
performance  on  the  main  task. 

6.5.3.3  Technology  Maturity,  Challenges,  and  Unresolved  Issues 

3D  perspective  displays  are  used  in  a  variety  of  applications.  Current  computing  and  graphic  processing 
power  allow  applicability  in  near  future  UAV  applications. 

Although  stereoscopic  displays  already  exist  in  several  forms,  their  application  is  still  limited.  The  two 
primary  bottlenecks  are  the  associated  lack  of  viewing  comfort  and  the  interference  with  other  tasks.  We  argue 
that  the  construction  of  “transparent  depth”  displays  is  an  important  new  development  because  it  does  not 
suffer  from  either  of  these  two  drawbacks.  We  believe  the  time  is  near  to  introduce  transparent  depth  displays 
in  professional  environments  like  the  cockpit,  command  and  control  workstations,  vehicles,  and  hand-held 
devices. 
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6.5.4  Spatial  Audio  Displays 
6.5.4.1  Description  of  Technology 

Providing  the  appropriate  information  to  support  situation  awareness  is  a  primary  challenge  in  the 
development  of  displays  and  controls  for  operators  in  any  complex  environment,  but  may  be  particularly 
challenging  for  the  designer  of  interfaces  for  UMVs.  A  pilot  in  a  traditional  manned  aircraft  can  directly 
perceive  elements  in  the  real  world  and  may  rapidly  develop  an  understanding  of  the  problem  space  by 
gleaning  ambient  information  from  peripheral  factors  including  weather,  terrain,  and  other  vehicles  in  the 
airspace;  he  or  she  can  maintain  some  level  of  understanding  of  the  general  vehicle  status  from  displays  in  the 
cockpit,  the  auditory  environment,  and  other  crew  members.  However,  many  of  these  real-world  cues  are  not 
as  readily  available  to  the  operator  of  a  UMV.  Thus,  the  use  of  effective  display  technologies  is  critical  to 
mission  success. 

Currently,  UAV  operator  interfaces  emphasize  the  presentation  of  information  through  visual  displays.  While 
often  the  most  appropriate  means  for  information  conveyance,  such  systems  run  the  risk  of  overloading  the 
visual  information  processing  capacity  of  the  operator.  The  integration  of  display  technologies  focusing  on 
other  modalities  affords  the  potential  to  reduce  visual  workload  and  display  redundant  cueing  to  provide  for 
safeguards  against  undetected  or  unrecognized  operationally  meaningful  information  while  also  allowing  for 
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synergistic  relations  to  occur  for  the  conveyance  of  higher-order  information.  Auditory  display  technologies, 
in  particular,  have  shown  great  promise  both  as  an  information-bearing  channel  in  isolation  and  as  a 
component  of  an  overall  multi-modal  display  system.  Audition  excels  as  an  early  warning  system  -  sound  is 
inherently  interpreted  with  respect  to  its  signalling  or  warning  significance.  In  addition,  neural  transmission  in 
the  auditory  pathway  is  superior  to  that  in  the  visual  system,  making  it  ideal  for  the  display  of  time-critical 
warnings.  The  auditory  system  also  plays  a  fundamental  role  in  verbal  communication,  which  is  in  many  cases 
the  most  direct,  efficient,  and  unambiguous  means  of  information  transfer.  What  is  inherently  appealing  about 
these  characteristics  of  the  auditory  system  is  that  they  are  attention-demanding  and  serve  to  make  the 
individual  immediately  aware  of  relevant  elements  in  the  situation.  Moreover,  the  auditory  system  provides 
this  information  independent  of  the  location  of  the  event,  for  the  auditory  system  monitors  the  environment 
from  all  directions  at  all  times.  Thus,  information  can  be  obtained  about  events  in  the  environment  even  when 
they  occur  outside  of  the  operator’s  visual  field-of-view. 

Although  auditory  displays  do  exist  in  most  operational  interfaces,  they  are  rudimentary  at  best,  and  fail  to 
leverage  the  natural  spatial  auditory  processing  capabilities  of  humans.  That  is,  the  ability  of  humans  to 
determine  the  location  of  a  sound  source,  monitor  events  at  multiple  locations  simultaneously,  and  utilize 
auditory  space  as  a  means  of  organizing  information,  has  not  been  fully  exploited.  Spatial  auditory  display 
technologies  take  advantage  of  the  properties  of  the  binaural  auditory  system  by  recreating  and  presenting  to 
an  operator  the  spatial  information  that  would  naturally  be  available  in  a  “real-world”  listening  environment. 
Such  displays  are  intuitive;  they  require  little  training  and  impose  few  additional  demands  on  the  information 
processing  capacity  of  the  operator. 

The  basic  approach  to  generating  spatialized  (virtual)  auditory  displays  assumes  a  “principle  of  equivalence”  - 
that  is,  given  that  the  display  can  generate  a  sound  field  at  the  eardrum  that  is  identical  to  the  sound  field  that 
would  result  from  a  real  source,  the  perception  should  be  equivalent  to  that  for  a  real  source.  Thus,  a  virtual 
auditory  display  will  result  in  the  perception  of  sounds  originating  from  real  locations  in  space  external  to  the 
listener’s  head. 

The  generation  of  virtual  auditory  displays  is  possible  because  of  what  is  understood  about  the  underlying 
mechanisms  of  spatial  hearing  and  the  cues  that  mediate  sound  localization.  These  cues  arise  as  a  result  of  the 
direction-specific,  frequency-dependent  modifications  that  are  imposed  on  an  incident  waveform  by  a 
listener’s  head,  torso,  and  pinnae  as  the  sound  travels  from  a  source  to  a  listener’s  eardrums.  Two  of  these 
cues,  interaural  time  differences  (ITDs)  and  interaural  level  differences  (ILDs),  mediate  sound  localization  in 
the  left/right  dimension.  ITDs  result  from  the  fact  that,  due  to  path  length  differences,  an  acoustic  waveform 
will  arrive  at  the  ipsilateral  ear  (i.e.,  ear  nearest  the  sound  source)  earlier  than  at  the  contralateral  ear 
(i.e.,  the  ear  on  the  side  of  the  head  opposite  the  sound  source),  leading  to  an  interaural  difference  in  time  of 
arrival.  Although  these  resulting  interaural  differences  are  small  (<  800  ps),  they  are  large  relative  to  the 
temporal  sensitivity  of  the  auditory  system,  which  can  detect  ITDs  on  the  order  of  10  ps  [1].  Because  of 
neural  processing  constraints,  however,  ITDs  are  primarily  useful  only  in  the  low-frequency  region 
(i.e.,  below  about  1.5  kHz).  For  frequencies  above  approximately  3.0  kHz  (a  region  in  which  the  wavelength 
of  an  acoustic  stimulus  is  small  relative  to  the  size  a  listener’s  head),  the  head  casts  an  acoustic  shadow  such 
that  the  stimulus  level  at  the  contralateral  ear  is  attenuated  relative  to  the  level  at  the  ipsilateral  ear. 
The  resulting  ILDs  are  useful  for  localizing  sounds  in  this  frequency  region  [2]. 

The  interaural  differences  described  above  provide  robust  cues  for  sound  localization  in  the  left/right 
dimension;  however,  they  are  not  sufficient  to  account  for  localization  in  elevation  or  front/back 
discrimination.  Indeed,  there  are  sets  of  locations,  known  as  cones  of  confusion,  which  all  produce  the  same 
interaural  difference  cues.  A  common  result  is  that  sounds  will  be  localized  to  the  wrong  position 
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(e.g.,  the  wrong  elevation  or  front/back  position)  on  the  correct  cone  of  confusion.  The  shape  of  the  stimulus 
spectrum  that  reaches  a  listener’s  ears  can  help  a  listener  disambiguate  the  location  of  a  sound  falling  along  a 
cone  of  confusion  by  providing  cues  regarding  the  elevation  and  front/back  position  of  the  sound.  Narrowband 
spectral  peaks  and  notches  emerge  from  interactions  between  the  stimulus  and  the  pinnae  in  the  frequency 
region  above  approximately  5.0  kHz  (where  the  wavelength  of  the  stimulus  is  small  relative  to  the  size  of  the 
pinna  structures).  These  peaks  and  notches  vary  in  relatively  systematic  ways  as  a  function  of  elevation  of  the 
sound  source,  thus  providing  cues  for  localization  in  this  dimension  [3,4,5].  Additionally,  each  pinna  casts  an 
acoustic  shadow  such  that  a  sound  originating  from  a  listener’s  rear  hemifield  will  have  relatively  less  energy 
in  the  3.0  kHz  to  6.0  kHz  region  than  a  sound  originating  in  the  frontal  hemifield  [6].  This  filtering  is 
presumed  to  help  a  listener  resolve  front/back  confusions  [7,8]. 

One  additional  cue  that  contributes  to  a  listener’s  ability  to  determine  the  front/back  location  of  a  sound  source 
arises  from  the  fact  that  interaural  difference  cues  change  in  predictable  ways  with  head  movement. 
For  example,  given  a  sound  in  the  frontal  hemifield,  a  clockwise  rotation  of  the  head  will  lead  to  ITDs  and 
ILDs  that  indicate  a  sound  moving  to  the  left  relative  to  the  listener;  a  counter  clockwise  rotation  in  response 
to  this  same  source  will  lead  to  the  opposite  experience.  Listeners  use  these  exploratory  head  movements  to 
disambiguate  the  front/back  location  of  a  sound  source. 

All  of  these  transformations  that  a  waveform  undergoes  as  it  travels  from  a  source  to  a  listener’s  eardrums  can 
be  measured  and  captured  in  what  is  known  as  the  head-related  transfer  function  (HRTF;  [9]).  HRTFs  can  be 
measured  by  placing  microphones  in  the  left  and  right  ear  canals  of  a  listener  (or  a  mannequin  head) 
and  making  recordings  of  broadband  acoustic  stimuli  presented  from  a  number  of  directions.  Digital  filters  are 
constructed  from  these  HRTFs  and  a  virtual  auditory  stimulus  can  then  be  generated  by  convolving  an 
arbitrary  signal  with  the  HRTFs  for  the  left  and  right  ear  associated  with  a  specific  position  in  space. 
The  result,  when  played  back  over  headphones,  is  an  experience  in  which  the  listener  perceives  the  sound  to 
have  originated  from  that  particular  position  in  space.  When  done  correctly,  such  a  display  can  support  sound 
localization  performance  that  is  equivalent  to  the  localization  of  real  sources  in  the  free  field  [10].  Note  that 
while  virtual  auditory  displays  can  be  delivered  via  loudspeaker  systems  instead  of  headphones, 
the  veridicality  of  such  displays  is  constrained  by  the  size  and  acoustic  characteristics  of  the  room  in  which 
the  display  is  presented.  In  addition,  there  is  a  relatively  small  region  of  space  within  a  loudspeaker  array  over 
which  a  virtual  auditory  display  will  be  valid  for  a  listener.  Thus,  it  is  generally  believed  that  headphone 
delivery  of  such  displays  is  best.  Such  a  practice  allows  the  display  designer  to  have  complete  control  over  the 
signal  arriving  at  the  listener’s  ears,  independent  of  the  conditions  in  which  the  display  is  presented. 

In  many  situations,  the  spatial  auditory  display  is  required  to  present  sounds  that  remain  fixed  in  virtual  space, 
independent  of  a  listener’s  head  movement.  To  accomplish  this,  the  filters  used  to  generate  the  virtual 
stimulus  must  be  updated  (in  real-time)  in  response  to  the  listener’s  head  movement.  This  is  achieved  by 
continuously  measuring  the  position  of  the  listener’s  head  using  a  head  tracking  system  and  providing  this 
position  information  to  the  system  responsible  for  rendering  the  spatial  auditory  stimuli.  This  system  then 
convolves  a  sound  with  the  filter  associated  with  the  appropriate  location  relative  to  the  orientation  of  the 
listener  at  each  point  in  time.  Furthermore,  because  the  HRTFs  define  a  constrained  set  of  discrete  locations 
from  which  a  virtual  stimulus  can  be  presented,  an  interactive  display  that  includes  head  movement  must  be 
able  to  interpolate  between  sets  of  spatially  adjacent  HRTFs  in  order  to  present  virtual  stimuli  from  any 
arbitrary  location. 

A  spatial  auditory  display  with  head  tracking  has  a  number  of  advantages.  First,  as  discussed  earlier,  head 
movements  can  help  a  listener  disambiguate  the  location  of  a  sound  source  in  the  front/back  dimension, 
thus  improving  localization  performance.  Similarly,  such  a  display  allows  the  listener  to  orient  toward  a 
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specific  sound,  bringing  it  into  the  “auditory  fovea,”  where  spatial  acuity  is  greatest.  However,  one  of  the 
greatest  advantages  of  an  interactive  display  is  that  it  can  be  substantially  more  immersive  than  a  static  display 
because  the  auditory  environment  reacts  in  realistic  and  expected  ways  in  response  to  a  listener’s  head 
movements.  An  interactive  display  is,  in  fact,  necessary  if  one  wishes  to  use  virtual  auditory  stimuli  to 
indicate  the  location  of  objects  (such  as  targets  and  threats,  as  described  below)  in  real  space. 

6.5.4.2  Actual  or  Potential  Application  to  UMVs 

As  previously  stated,  the  development  of  displays  to  support  situation  awareness  must  be  a  primary  goal  of  the 
designer  for  UMV  operator  interfaces.  Situation  awareness  (SA)  has  been  defined  as  “the  perception  of  the 
elements  in  the  environment  within  a  volume  of  time  and  space,  the  comprehension  of  their  meaning  and  the 
projection  of  their  status  in  the  near  future”  [11].  From  this,  it  has  been  suggested  that  SA  may  be  more 
formally  broken  down  into  three  component  levels  that  can  be  independently  addressed,  studied, 
and  measured.  They  are:  Level  1  SA,  which  refers  to  the  perception  of  elements  in  an  environment  within  a 
particular  volume  of  time  and  space;  Level  2  SA,  which  pertains  to  the  comprehension  of  the  meaning  of  these 
elements;  and  Level  3  SA,  which  is  concerned  with  the  projection  of  the  status  of  the  elements  in  the  near 
future. 

If  one  considers  the  types  of  SA-related  errors  that  occur  in  aviation  mishaps  within  the  context  of  the  UMV 
operator  environment,  it  is  clear  that  spatial  audio  may  offer  significant  utility  at  each  level  of  SA.  The  SA 
Error  Taxonomy  [12]  describes  several  reasons  that  situation  awareness  may  break  down  within  operational 
settings.  Level  1  SA  errors  have  been  shown  to  account  for  76%  of  aviation  mishaps  that  are  attributed  to 
human  error  [13].  These  errors  could  occur  as  a  result  of  the  lack  of  availability  of  required  data  or  a  failure  of 
the  system  to  present  available  data.  Errors  might  also  occur  because  the  data  are  provided  to  the  operator, 
but  are  difficult  to  detect  or  are  merely  not  observed,  not  attended  to,  misperceived,  or  forgotten  by  the 
operator.  Spatial  auditory  displays  would  likely  benefit  the  operator  in  each  of  these  situations. 

The  auditory  system’s  ability  to  serve  as  an  early  warning  system,  as  well  as  its  unique  capacity  for 
monitoring  all  locations  simultaneously,  can  be  exploited  in  spatial  auditory  displays.  The  auditory  system  is 
exquisitely  sensitive  to  change,  even  when  this  change  occurs  outside  of  the  current  focus  of  attention. 
Changes  associated  with  onsets  (e.g.,  the  introduction  of  new  elements  into  a  display)  and  offsets 
(e.g.,  the  removal  of  an  existing  element  from  a  display)  are  particularly  well-detected.  They  are,  in  fact, 
often  impossible  to  ignore,  and  drive  the  allocation  of  attention  to  elements  in  the  environment  that  may  yield 
critical  information,  thus  reducing  the  chance  that  this  information  will  go  undetected.  So  natural  is  this 
phenomenon  that  several  authors  have  suggested  that  the  spatial  auditory  system  evolved  specifically  to  direct 
the  visual  system  to  meaningful  events  in  the  environment.  Support  for  this  hypothesis  can  be  seen  in 
laboratory  studies  involving  visual  search  tasks,  where  spatially  coincident  auditory  cues  have  been  shown  to 
reduce  visual  target  acquisition  and  identification  times  by  a  factor  of  2-5  in  very  simple  visual  scenes;  much 
greater  benefits  have  been  found  in  more  complex  visual  scenes  such  as  those  depicted  in  simulated 
operational  environments  [14,15,16].  Moreover,  redundantly  coding  information  using  both  auditory  and 
visual  stimuli  may  help  to  overcome  systematic  misperceptions.  That  is,  the  display  of  auditory  information 
that  is  consistent  with,  and  covaries  with,  visual  information  is  not  only  unambiguous,  but  is  consistent  with 
operator  expectancies,  thus  providing  a  more  natural,  intuitive  interface.  Given  these  issues,  it  seems  clear  that 
spatial  audio  cueing  might  be  especially  useful  to  UMV  operators  who  are  tasked  with  finding  ground  targets 
in  a  remote  environment  through  the  control  of  a  maneuverable  camera. 

The  success  of  future  UMV  systems  will  also  depend  on  an  operator’s  ability  to  monitor  multiple  independent 
sources  of  information  in  order  to  maintain  situation  awareness.  Here,  again,  the  auditory  modality  is 
particularly  well  suited  to  the  task,  for  the  auditory  system  is  capable  of  segregating  multiple  simultaneous 
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sounds  into  different  streams  [17],  thus  allowing  an  observer  to  attend  to  the  most  relevant  information  at  any 
given  time  and  to  relegate  the  remaining  information  to  the  “background”  for  further  processing  when 
necessary.  This  ability  to  focus  on  certain  information,  while  simultaneously  and  in  parallel  monitoring  and 
maintaining  a  functional  level  of  awareness  about  other  sources  of  information,  is  invaluable  in  complex  and 
dynamic  operational  environments.  Because  this  segregation  process  is,  in  part,  mediated  by  space  [18],  it  can 
be  exploited  using  spatial  auditory  displays. 

One  environment  in  which  the  utility  of  spatial  audio  for  segregation  has  been  extensively  studied  is  that  of 
the  operator  who  must  monitor  multiple  simultaneous  communication  channels.  Here,  substantial 
improvements  in  workload  and  communication  effectiveness  can  be  seen  when  the  individual  communication 
channels  are  presented  such  that  they  appear  to  originate  from  different  locations  in  space,  unlike  standard 
monaural  communication  systems  [19,20,21].  The  flexibility  of  this  display  technology  allows  a  user  to 
configure  the  display  such  that  particular  communication  channels  are  assigned  to  specific  locations  that  are 
meaningful  to  the  user.  The  UMV  operator  who  must  engage  in  verbal  communications  with  a  variety  of 
distributed  team  members  could  make  use  of  such  a  display  to  maintain  an  awareness  of  the  identity  of  the 
specific  talker  from  which  each  communication  originated.  That  is,  the  operator  can  use  space  as  an 
organizing  principle,  and  this  may  result  in  enhancements  of  both  Level  1  and  Level  2  SA. 

The  appropriate  development  and  comprehension  of  mental  models  necessary  for  achieving  Level  2  SA  may 
also  be  supported  by  an  auditory  environment  in  which  the  operator  is  immersed  and  experiences  a  sense  of 
presence  (i.e.,  “being  there”).  Support  for  this  comes  from  the  work  of  Ramsdell,  who  suggested  that  one 
function  of  the  auditory  system  is  to  connect  one  to  the  real  world  on  a  primitive  level,  utilizing  the  incidental 
sounds  that  serve  to  make  up  the  auditory  “background.”  [22].  This  work  was  based  on  reports  from  suddenly- 
deafened  individuals  who  reported  that  the  world  seemed  “dead”  and  “(un)coupled,”  that  “it  was  almost 
impossible  to  believe  in  the  passage  of  time  . .  .couldn’t  hear  a  clock  tick”  (p.503).  Ramsdell  distinguished  this 
level  of  auditory  perception  from  that  of  communication  and  warning,  which  are  more  obvious  and  overt 
functions  of  the  auditory  system,  and  suggested  that  this  primitive  level  of  perception  is  critical  for  a  sense  of 
“connectedness”  to  the  world.  This  experience  of  suddenly-deafened  individuals  has  been  likened  to  that  of 
the  user  of  a  virtual  environment  with  an  impoverished  auditory  display  [23].  Perceptually  rich  virtual 
auditory  environments  are  believed  to  lead  to  a  strong  sense  of  presence  [24].  Although  the  link  between 
presence  and  task  performance  is  less  clear  [25]  than  that  believed  to  exist  between  situation  awareness  and 
performance,  it  has  been  suggested  that  this  is  due  in  part  to  the  lack  of  a  robust  measure  of  presence  and/or 
the  use  of  gross  performance  metrics  that  may  not  be  sensitive  to  issues  regarding  how  the  interface  is  actually 
being  used  [26],  and  thus  how  the  sense  of  presence  may  contribute  to  that  usage.  The  sense  of  presence  is 
concomitant  with  an  engagement  on  the  part  of  the  operator,  and  this  may  be  critical  when  the  operator  takes 
on  a  supervisory  role  over  semi-autonomous  UMVs.  In  this  situation,  there  exists  the  potential  that  the 
operator  will  ‘fall  out’  of  the  control  loop  and  may  have  difficulty  re-entering  when  necessary.  Immersion  in 
the  virtual  environment  (i.e.,  the  UMV  operator  interface)  may  facilitate  intuitive  interaction  and  ensure  that 
the  operator  remains  engaged  in  the  mission  even  if  not  directly  flying  the  vehicle. 

Finally,  the  support  of  Level  3  SA  may  be  assisted  by  an  auditory  display  that  is  spatially,  spectrally, 
and  temporally  dynamic.  Information  about  the  current  and  future  states  of  highly-dimensional  environments 
may  be  related  via  auditory  information  in  a  way  that  is  engaging  and  intuitive.  Operators  may  discern  overall 
relationships  and  trends  in  order  to  better  predict  future  states  [27].  Auditory  motion  perception  can  be  used  to 
demonstrate  trajectories  of  elements  in  the  environment  and  are  particularly  compelling  when  used  in 
conjunction  with  analogous  visual  displays  for  predicting  future  states,  allowing  the  UMV  operator  to 
“fly  several  seconds  ahead  of  the  aircraft.”  This  temporal  aspect  of  situation  awareness  may  be  particularly 
well-supported  by  a  spatial  auditory  display. 
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6.5.4.3  Technology  Maturity,  Challenges,  and  Unresolved  Issues 

Many  of  the  technological  challenges  that  were  considered  in  early  iterations  of  spatial  auditory  displays  have 
been  largely  overcome,  the  most  obvious  of  which  being  adequate  computational  power.  Increased  memory 
capacity  has  allowed  for  the  storage  of  a  large  number  of  measured  HRTF  locations,  thus  increasing  the 
resolution  of  the  rendered  auditory  space.  Advances  in  digital  signal  processing  capabilities  have  allowed  for 
the  rapid  convolution  of  stimuli  with  more  detailed  representations  of  the  HRTFs,  and  thus  the  presentation  of 
increasingly  complex  auditory  environments  (e.g.,  multiple  sources,  room  acoustics  characteristics)  that  have 
the  potential  to  yield  richer  and  more  compelling  experiences  for  UMV  operators. 

Perhaps  the  greatest  challenges  that  remain  in  the  generation  of  veridical  and  compelling  spatial  auditory 
displays  concern  the  psychoacoustic  questions  that  continue  to  be  of  interest  to  auditory  scientists.  One  issue 
concerns  the  fact  that  HRTFs  are  highly  individualized,  and  it  has  been  shown  that  the  localization  of  virtual 
auditory  stimuli  is  best  when  listening  through  one’s  own  HRTFs.  This  is  especially  true  for  localization  in 
those  dimensions  in  which  performance  is  mediated  by  spectral  cues  (i.e.,  elevation  and  front/back 
dimensions).  However,  because  it  is,  in  many  cases,  logistically  impossible  to  collect  HRTF  measurements  on 
each  individual  operator,  the  most  common  practice  involves  the  use  of  generic  HRTFs  that  were  originally 
measured  using  a  mannequin  head.  A  better  understanding  of  the  relative  importance  of  various  spectral 
features  for  localization,  and  how  these  features  are  recovered  by  the  auditory  system,  might  allow  for  the 
construction  of  an  effective  set  of  non-individualized  HRTFs.  Such  HRTFs  could  improve  performance  over 
existing  generic  models  of  auditory  space  and  perhaps  yield  localization  performance  comparable  to  that 
found  for  free-fleld  stimuli  (i.e.,  greater  accuracy,  fewer  front/back  confusions,  greater  externalization  of  the 
auditory  images). 

Another  issue  that  must  be  addressed  in  future  developments  of  spatial  auditory  displays  is  the  encoding  of 
veridical  sound  source  distance.  For  an  arbitrary  stimulus,  there  exist  several  acoustic  cues  that  yield  source 
distance  information,  among  them  sound  source  intensity,  gross  spectral  characteristics,  and  the  ratio  of  direct- 
to-reverberant  acoustic  energy  reaching  a  listener  [28,29].  It  has  been  suggested  that  listeners  utilize  these 
cues  when  determining  the  distance  of  a  remote  sound  source,  thus  dictating  that  such  cues  be  incorporated 
into  displays  lest  one  provide  impoverished  distance  information.  In  addition,  in  the  auditory  near  field 
(i.e.,  less  than  1  m  from  the  head),  distance  perception  is  mediated  in  large  part  by  characteristics  in  the  HRTF 
that  vary  with  source  distance,  in  particular  interaural  level  differences  [30].  These  cues  appear  to  provide 
relatively  salient  cues  about  sound  source  distance,  and  thus  should  be  implemented  in  future  displays. 
However,  auditory  displays  are  likely  to  be  used  in  noisy  environments,  and  noise  serves  to  disrupt 
localization  cues  not  only  in  azimuth  and  elevation  [31],  but  can  also  disrupt  the  cues  for  source  distance  by 
masking  the  reverberation  and  spectral  cues  in  the  display,  and  by  reducing  the  dynamic  range  utilized  for 
intensity-based  distance  coding.  New  symbologies  must  be  developed  that  could  be  used  in  conjunction  with 
existing  cues  to  provide  more  reliable  sound  source  distance.  For  example,  one  display  that  has  been  proposed 
[32]  utilizes  the  effort  with  which  a  speech  phrase  is  spoken  to  indicate  distance  -  shouted  speech  indicates 
sources  at  greater  distances,  whispered  speech  indicates  closer  sources,  and  conversational  speech  indicates 
sources  at  distances  in  between. 

It  is  important  to  note  that  spatial  auditory  displays  have  been  integrated  and  tested  in  a  number  of  operational 
environments  including  fighter  jets  and  general  aviation  aircraft,  as  well  as  command  and  control  centers 
[33,34].  The  Air  Force  Research  Laboratory  has  implemented  a  spatial  audio  display  in  two  ground-based 
controller  stations  as  part  of  a  communications  system  upgrade  at  a  training  facility.  This  system,  which 
allows  users  to  allocate  incoming  voice  communications  to  seven  different  apparent  locations,  has  made  it 
possible  for  dozens  of  operators  to  experience  the  advantages  of  spatial  audio  during  the  conduct  of  realistic 
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training  missions.  Operator  feedback  indicates  that  these  spatial  auditory  displays  allowed  the  operators  to 
have  “total  SA”  in  the  mission.  Their  comments  indicate  not  only  overwhelming  acceptance  by  the  user 
community,  but  also  the  increase  in  mission  effectiveness  that  is  possible  due  to  the  enhanced  situation 
awareness  supported  by  spatial  audio.  Spatial  audio  can  therefore  be  classified  as  having  a  Technology 
Readiness  Level  of  8. 

There  are  a  number  of  unexplored  applications  for  spatial  audio  that  have  the  potential  to  enhance  situation 
awareness  for  UMV  operators.  The  need  to  monitor  multiple  simultaneous  environments  (e.g.,  the  virtual 
operational  environment  and  the  real-world  environment  in  which  the  operator  station  is  located)  may  be 
supported  by  signal  processing  techniques  employing  room  acoustics  models  to  make  the  two  categories  of 
display  elements  appear  to  originate  from  different  “rooms.”  An  auditory  environment  that  is  slaved  to  the 
UMV  camera  may  allow  the  operator  to  unambiguously  center  a  visual  target  in  a  complex  visual  scene  that 
would  otherwise  be  difficult  to  find.  Spatial  audio  displays  can  also  lead  to  a  level  of  realism  that  as  yet  cannot 
be  achieved  in  visual  displays.  Thus,  they  contribute  substantially  to  a  sense  of  presence  and  task  engagement 
that  could  potentially  improve  overall  operator  performance.  The  challenge  is  to  identify  those  specific 
features  that  contribute  to  presence  and  implement  them.  Nevertheless,  in  the  nearly  20  years  since  the  first 
functional  spatial  auditory  display  system  was  introduced,  great  advances  have  been  made  in  both  science  and 
technology,  resulting  in  a  display  that  is  reliable,  mature,  and  cost-effective. 
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6.5.5  Haptic  Display  Technology 
6.5.5.1  Description  of  Technology 

Designers  of  human  machine  interfaces  are  increasingly  applying  multi-modal  interfaces.  An  important  reason 
for  this  is  the  need  for  an  alternative  or  complementary  information  channel  in  complex  operator 
environments  [1,2,3].  Traditionally,  the  auditory  channel  is  often  considered  as  an  alternative  or  supplement  to 
visual  displays.  Examples  include  the  presentation  of  route  navigation  [4,5]  and  tracking  error  information 
[6,7].  However,  there  are  situations  in  which  the  visual  and  auditory  channels  of  an  operator  are  both  heavily 
loaded  or  in  which  the  visual  and/or  auditory  information  is  degraded.  In  those  situations,  a  haptic  display 
system  may  be  useful. 

An  example  of  a  tactile  display  is  the  TNO  tactile  waist  belt  (Figure  6-26).  The  tactile  display  consists  of  eight 
vibrating  elements  (1.3  V  vibrating  DC  motors,  housed  in  rectangular  PVC  boxes)  with  a  body  contact  area  of 

1.5  by  2.0  cm  and  a  vibration  frequency  of  155  Hz.  The  boxes  are  mounted  in  an  adjustable  waist  belt. 
The  resolution  of  the  displays  (i.e.,  8  tactors  for  360°)  is  in  between  the  minimum  required  (i.e.,  two  elements: 
one  for  left  and  one  for  right)  and  the  limit  of  direction  perception  on  the  torso  (to  be  in  the  order  of  10°). 
The  location  of  the  elements  in  the  belt  is  adjustable  so  they  can  easily  be  positioned  in  the  direction  of  the 
cardinal  and  oblique  axes  irrespective  of  the  body  form  of  the  person  who  is  wearing  the  belt.  The  waist  belt  is 
worn  over  the  underclothing  of  the  person. 
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Figure  6-26:  Example  Tactile  Display  (Tactile  Waist  Belt  Clearly  Showing  the  Vibration  Elements). 


Another  example  is  the  tactile  torso  display.  TNO  [8,9]  used  a  torso  display  consisting  of  64  vibro-tactile 
elements  that  presented  information  concerning  the  desired  direction  of  motion  (simple  version),  and  a  torso 
display  that  included  information  concerning  an  actual  motion  direction  (complex  version).  Figure  6-27  shows 
such  a  torso  display  used  in  experiments  based  on  helicopter  scenarios. 


Figure  6-27:  Example  of  a  Tactile  Torso  Display  Used  in  Helicopter  Orientation  Studies  [21]. 


6.5.5.2  Actual  or  Potential  Application  to  UMVs 

6. 5. 5. 2. 1  Vibrotactile  Displays 

Vibrotactile  displays  present  information  by  delivering  a  localised  vibration  to  the  skin.  Partly  due  to  the 
trends  in  multi-modal  interfaces,  this  kind  of  display  is  going  through  a  rapid  development.  Although  the 
technology  to  build  active  displays  was  already  developed  40  years  ago,  the  applications  were  mainly 
restricted  to  research  tools.  An  important  example  is  the  TVSS,  a  device  developed  by  Bach-y-Rita  and 
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colleagues  that  displays  visual  information  acquired  by  a  camera  on  a  144-element  vibrotactile  display 
attached  to  the  abdomen  of  the  observer  [10].  Today,  cellular  phones  with  vibration  function  are  probably  the 
best-known  example  of  a  tactile  display,  but  many  examples  of  tactile  displays  are  also  developed  within  the 
military  domain.  In  relation  to  UAVs,  tactile  displays  are  receiving  a  small  share  of  interest,  as  far  as  we  know 
they  are  only  investigated  in  several  laboratories,  but  are  not  on  the  market  yet. 

We  will  discern  the  following  two  areas  in  which  adding  a  tactile  display  to  the  UMV-operator  interface  may 
be  beneficial. 

a)  (Directional)  warning/attention  allocation  system.  The  vibration  function  on  a  mobile  phone  and  the 
stick  shaker,  which  warns  pilots  that  the  aircraft  is  in  danger  of  stalling,  are  two  examples  of  a  tactile 
warning  system.  Experimental  evidence  that  indicates  that  vibrotactile  displays  are  well  suited  to 
detect  time  critical  events  is  provided  by  several  authors.  For  example,  Martens  and  Van  Winsum 
[11]  found  that  tactile  warning  cues  were  more  effective  than  speech  warning  cues  in  presenting 
collision  avoidance  warnings.  Sklar  and  Sarter  [12]  demonstrated  that  the  use  of  tactile  cues  for 
indicating  unexpected  changes  in  status  are  more  effective  than  visual  cues.  Additionally,  some 
research  has  been  conducted  on  the  usefulness  of  vibrotactile  displays  within  UAV  control 
applications  [13,14,15].  The  UAV  operator  wore  small  tactors  mounted  in  elastic  bands,  one  on  each 
inner  wrist  (Figure  6-28).  When  a  high  priority  system  contingency  occurred,  one  or  both  of  the 
tactors  vibrated.  By  noting  which  tactor(s)  was  vibrating,  the  operator  could  rapidly  identify  the 
system  that  needs  attention,  especially  when  attention  was  directed  to  a  display  not  containing  a  visual 
warning.  This  technology  has  strong  potential  for  directing  attention  to  unexpected  events  in  future 
UMV  control  environments  where  the  operator  supervises  automated  systems. 


Figure  6-28:  Tactile  Wrist  Pads  as  High  Priority  Alert  Cue. 

b)  Spatial  information.  This  category  is  of  special  interest  to  UMV  applications  and  also  the  area  that 
receives  attention  for  military  applications.  Since  the  nineties,  the  tap-on-the-shoulder  principle  is 
implemented  in  tactile  torso  displays.  The  power  behind  this  concept  is  that  the  proverbial  tap  on 
shoulder  draws  and  directs  the  spatial  attention  of  the  observer.  Displays  that  can  provide  this  tap  on 
any  location  on  the  torso  and  that  consider  the  torso  as  a  3D  sphere  are  able  to  project  3D  spatial 
information  directly  and  intuitively.  Several  preconditions  for  successful  application  of  this  principle 
have  been  fulfilled,  e.g.,  the  spatial  resolution  and  the  accuracy  of  direction  perception  have  been 
investigated  [16],  as  are  the  effects  of  applying  the  principle  in  dynamic  environments  and  in 
combination  with  other  sensory  modalities  [17,18].  Proof-of-concept  studies  also  show  encouraging 
results  for  helicopter  hover  tasks  [19,20,21],  spatial  disorientation  situations  [22,23,24],  waypoint 
navigation  [16]  and  vehicle  control  [24,25].  In  UMVs,  acquiring  spatial  information  or  a  good  sense 
of  spatial  awareness  is  one  of  the  critical  issues.  For  example,  the  multiple  frames  of  reference  for  an 
operator  (especially  a  moving  operator,  i.e.,  an  operator  that  is  on  board  a  moving  platform)  may  slow 
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down  this  process  and  the  quality  of  the  decisions  made.  A  tactile  display  that  uses  the  tap-on-the- 
shoulder  principle  can  be  a  powerful  situation  awareness  support  [26,27,28],  for  instance  by 
presenting  the  heading  direction  of  the  vehicle. 

6. 5. 5. 2. 2  Force  Feedback  Displays 

Force  feedback  (proprioceptive)  displays  present  information  via  forces  on  the  controls.  Force  feedback 
devices  can  be  used  in  a  general  human-computer  interaction  setting  (e.g.,  a  force  feedback  mouse  or 
rollerball),  but  are  not  particularly  widespread.  On  the  other  hand,  force  feedback  joysticks  and  steering 
wheels  are  quite  common  in  gaming.  Also,  force  displays  have  proven  their  value  in  many  remote  control 
situations.  In  UMVs,  force  feedback  devices  can  be  also  be  of  use,  amongst  others  because  the  control  of  their 
inhabited  equivalents  relies  on  these  devices.  For  example,  in  road  vehicles,  forces  (and  vibrations)  in  the 
steering  wheel  provide  important  information  about  the  road  conditions  as  well  as  the  vehicle  behaviour. 
The  same  holds  for  forces  in  flight  controls  of  helicopters  and  airplanes.  Especially  when  the  UMV  is 
controlled  in  the  loop,  this  information  may  be  of  great  value.  Force  displays  may  also  assist  operators  in 
camera  control  tasks,  for  example  by  presenting  platform  motions  via  joystick  motions  or  forces  [29,30]. 

Wind  turbulence  is  potentially  detrimental  to  safe  and  effective  UAV  tele-operated  control.  Unfortunately, 
the  physical  separation  of  the  crew  from  the  aircraft  makes  detection  of  sudden  turbulence  onset  very  difficult, 
often  solely  indicated  by  an  unexpected  perturbation  of  video  images  transmitted  from  a  UAV  mounted 
camera.  One  study  [31]  explored  how  to  provide  the  UAV  pilot  with  an  enhanced  indication  of  turbulence. 
Four  different  alerts  were  evaluated:  Visual  (perturbation  of  nose-camera  imagery  and  overlaid  HUD 
symbology  -  Baseline),  Visual/Haptic  (Visual  and  additional  1  second,  low  gain,  high  frequency  vibration  of 
the  control  stick  using  an  Immersion  Corporation  2000  Force  Feedback  Joystick),  Visual/ Aural  (Visual  and 
1  second  pure  tone),  Visual/ Aural/Haptic  (all  three  cues  simultaneously).  Data  were  collected  from  pilots  as 
they  performed  simulated  landing  tasks.  Conditions  containing  the  haptic  cue  (Visual/Haptic  and  Visual/ 
Haptic/ Aural)  resulted  in  less  error  than  non-haptic  cue  conditions  (Visual  and  Visual/ Aural).  Although  the 
aural  alert  also  improved  landing  accuracy  and  detection  of  turbulence  direction,  performance  was  best  with 
the  redundant  kinesthetic  feedback.  When  randomly  queried  regarding  the  primary  direction  of  the  UAV 
immediately  following  a  turbulence  event,  participants  were  more  accurate  when  haptic  feedback  had  been 
present.  It  should  be  noted  that  the  operators  commented  that  only  a  slight  haptic  feedback  is  required 
(and  desired)  to  alert  turbulence  onset. 

6.5.5.3  Technology  Maturity,  Challenges,  and  Unresolved  Issues 

The  technology  for  haptic  feedback  is  very  mature,  though  experiments  designed  to  determine  the 
effectiveness  of  haptic  information  for  military  applications  are  on-going.  Early  studies  show  promise  for  the 
effective  use  of  this  technology,  with  practical  military  applications  on  the  near  horizon. 
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6.6  INTERFACE  ISSUES  FOR  MULTI-UMV  SUPERVISORY  CONTROL 

6.6.1  Implications  of  Automation/Autonomy 

6.6.1. 1  What  is  Automation  and  Supervisory  Control? 

A  common  frame  of  reference  is  needed  to  discuss  automation  and  subsequently  supervisory  control. 
The  Oxford  Dictionary  of  Current  English  [1]  defines  automation  as  “use  or  introduction  of  automatic 
methods  or  equipment  in  place  of  manual  labour.”  Thus,  automation  takes  place  when  a  task  that  is  usually 
performed  by  a  human  is  performed  by  a  machine  (often  a  computer).  An  automatic  transmission  on  a  car, 
for  example,  takes  the  place  of  a  human  physically  shifting  the  gears,  and  automates  this  task.  The  process  by 
which  the  human  controls  the  automated  system  (i.e.,  selecting  the  appropriate  gear)  is  supervisory  control  [2]. 
This  section  will  focus  on  the  interaction  between  automation  and  supervisory  control  or  more  specifically, 
the  interface  implications  posed  by  automation. 

Automation  is  not  all  or  none.  It  comes  in  many  forms  and  varieties  and  has  been  characterized  as  levels  of 
automation  or  supervisory  control.  Sheridan  and  Verplank  [3]  proposed  one  of  the  first  automation 
taxonomies  characterized  by  eight  levels  of  supervisory  control  (Table  6-1). 


Table  6-1 :  A  Scale  of  Degrees  of  Automation  (Sheridan  and  Verplank,  1978) 


1 

The  computer  offers  no  assistance;  the  human  must  do  it  all. 

2 

The  computer  suggests  alternative  ways  to  do  the  task. 

3 

The  computer  selects  one  way  to  do  the  task  and  executes  that  suggestion, 

4 

if  the  human  approves,  or 

5 

allows  the  human  a  restricted  time  to  veto  before  automatic  execution,  or 

6 

executes  automatically,  then  necessarily  informs  the  human,  or 

7 

executes  automatically,  then  informs  the  human  only  if  asked. 

8 

The  computer  selects  the  method,  executes  the  task  and  ignores  the  human. 

More  recently  Parasuraman,  Sheridan  and  Wickens  [4]  developed  a  framework  for  making  decisions  on  what 
functions  should  be  automated  and  to  what  extent.  Four  classes  of  function  were  proposed: 

1)  Information  acquisition; 

2)  Information  analysis; 

3)  Decision  and  action  selection;  and 

4)  Action  implementation. 

Automation  can  vary  within  each  function  from  low  to  high  (see  Table  6-2)  and  a  particular  system  can  be 
automated  at  different  levels  in  each  of  the  four  functions.  The  model  then  presents  a  methodology  for 
deciding  what  level  of  automation  should  be  assigned  what  level  of  automation  for  a  particular  system. 
This  methodology  is  based  on  primary  criteria;  mental  workload,  situation  awareness,  and  complacency, 
as  well  as  secondary  criteria:  automation  reliability  and  costs  of  decisions/outcomes. 
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Table  6-2:  Levels  of  Automation  of  Decision  and  Action  Selection  [4] 


HIGH 

10 

The  computer  decides  everything,  acts  autonomously,  ignoring  the  human. 

9 

informs  the  human  only  if  it,  the  computer,  decides  to 

8 

informs  the  human  only  if  asked,  or 

7 

executes  automatically,  then  necessarily  informs  the  human,  and 

6 

allows  the  human  a  restricted  time  to  veto  before  automatic  execution,  or 

5 

executes  the  suggestion  if  the  human  approves,  or 

4 

suggests  one  alternative 

3 

narrows  the  selection  down  to  a  few,  or 

2 

The  computer  offers  a  complete  set  of  decision/  action  alternatives,  or 

LOW 

1 

The  computer  offers  no  assistance:  human  must  make  all  decisions  and  actions. 

6.6.1.2  Levels  of  Automation  Specific  to  UMVs 

These  general  taxonomies  of  automation  have  served  well  for  research  and  manufacturing  systems. 
The  unique  nature  of  UMVs,  though,  has  led  to  new  characterizations  of  levels  of  automation  and  the 
relationship  between  the  operator  and  the  UMV.  No  universally  agreed  upon  description  of  levels  of 
automation  has  emerged,  but  key  concepts  are  similar  in  different  taxonomies.  Two  examples  are  provided 
below.  The  U.S.  Office  of  the  Secretary  of  Defence  [5]  developed  a  UAV  roadmap  and  included  the  following 
levels  of  automation,  with  a  proposed  timeline  (Figure  6-29).  Note  that  an  exponential  increase  in  the  levels  of 
automation  is  predicted.  In  1995,  automation  was  at  level  2,  Real  Time  Health/Diagnosis,  progressing  to  level 
4,  Onboard  Route  Re-planning  by  2005.  However,  by  2015,  it  is  predicted  that  automation  will  reach  level  10, 
Fully  Autonomous  Swarms.  This  will  require  great  leaps  of  technology  and  perhaps  greater  leaps  of  our 
understanding  of  how  humans  interact  and  control  autonomous  systems. 
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Autonomous  Control  Levels 


Figure  6-29:  Levels  of  Automation  [5]. 


Another  conceptualisation  of  levels  of  automation  comes  from  the  U.S.  Army  Science  Board  [6] 
(Figure  6-30). 
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Figure  6-30:  Levels  of  Automation,  U.S.  Army  Science  Board  [6]. 


While  less  specific  about  time  frame,  the  rapid  increase  in  automation  is  also  evident  in  the  Army  Science 
Board  taxonomy. 

In  addition  to  the  various  levels  of  autonomy  taxonomies,  a  distinction  needs  to  be  made  concerning  the  levels 
of  interoperability  and  control.  These  levels  were  defined  in  a  recent  NATO  Standardisation  Agreement  or 
STANAG  [7],  as  outlined  in  Table  6-3. 


6-70 


RTO-TR-HFM-078 


dMNATO 
WP  I  OTAN 


ADVANCED  UMV  OPERATOR  INTERFACES 


Table  6-3:  Levels  of  Interoperability 


Level  I 
Level  II 

Level  III 
Level  IV 
Level  V 


Indirect  receipt  of  imagery  and/or  data. 

Direct  receipt  of  imagery  and/or  data;  where  direct  covers  of  reception  of  the  UAV  payload 
data  by  the  UCS  when  it  has  direct  line-of-sight  with  the  UAV  or  a  relay  device  which  has 
direct  line-of-sight  with  the  UAV. 

Control  and  monitoring  of  the  UAV  payload  in  addition  to  direct  receipt  of  imagery/data. 
Control  and  monitoring  of  the  UAV,  less  launch  and  recovery. 

Control  and  monitoring  of  the  UAV  (Level  IV),  plus  launch  and  recovery  functions. 


A  common  feature  in  the  levels  of  automation  taxonomies  is  that  the  level  of  autonomy  is  predicted  to  rise 
dramatically  over  the  next  several  years.  This  rise  in  autonomy  will  dramatically  affect  the  relationship  of  the 
operator  to  the  aircraft.  An  interface  designer  needs  to  carefully  examine  the  required  or  optimal  levels  of 
autonomy  and  the  levels  of  control.  The  rest  of  this  chapter  addresses  the  implications  of  this  rising 
automation  level. 


6.6.1.3  Automation:  A  Double-Edged  Sword 

6. 6. 1.3. 1  Potential  Benefits  of  Automation 

Why  do  we  automate?  There  are  several  benefits,  both  real  and  perceived.  However,  there  are  also  costs 
(or  potential  costs)  to  automation  and  this  section  will  address  these  costs  and  benefits. 

6. 6. 1.3. 1.1  Mission  Capability 

A  commonly  cited  benefit  of  using  UMVs  is  that  they  can  do  the  jobs  that  humans  are  either  reluctant  to  do  or 
cannot  do.  These  missions  have  been  referred  to  as  “dirty,  dangerous,  and  dull”.  UMVs  can  clean  up  toxic 
spills,  or  otherwise  dirty  environments  without  exposing  human  operators  to  this  potential  danger. 
UMVs  might  have  saved  lives  at  Chernobyl.  About  200,000  people  (“liquidators”)  from  all  over  the  USSR 
were  involved  in  the  recovery  and  clean  up  of  this  accident  during  1986  and  1987.  They  received  high  doses 
of  radiation,  around  100  millisieverts.  Some  20,000  of  them  received  about  250  mSv  and  a  few  received 
500  mSv.  Later,  the  number  of  liquidators  swelled  to  over  600,000,  but  most  of  these  received  only  low 
radiation  doses  [8].  UMVs  might  have  saved  lives  by  performing  this  “dirty”  mission.  UMVs  can  also  operate 
in  combat  environments  without  exposing  the  operators  to  danger.  Some  visions  of  the  future  suggest  that 
manned  war  as  we  know  it  won’t  exist  in  the  future.  We  will  instead,  see  battles  of  drone  aircraft  and  land 
vehicles.  Automated  vehicles  are  also  very  useful  for  performing  the  very  dull,  boring  missions.  These  might 
include  long  duration  surveillance  or  long  egress  segments.  UAVs  are  making  strides  in  development  of  this 
type  of  capability.  Within  two  years,  solar-electric  airplanes  incorporating  energy  storage  for  night-time 
operations  will  be  capable  of  continuous  flight  for  up  to  six  months  at  a  time  at  altitudes  over  60,000  feet. 
Applications  for  such  aircraft  include  telecommunications,  remote  sensing  and  atmospheric  measurement. 

6.6. 1 .3 . 1 .2  Affordability 

A  major  driver  in  automation  is  the  perception  that  automated  systems  are  less  expensive  to  build  and  operate 
than  manned  systems.  This  is  especially  true  in  the  case  of  small  UMVs.  Small  UAVs  such  as  the  Raven  and 
Pointer  can  be  fielded  inexpensively  with  non-pilot  personnel  trained  to  operate  them,  further  reducing  the 
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operational  costs.  Since  the  initial  cost  is  low,  these  assets  can  be  considered  attritable.  That  is,  loss  of  the 
asset  does  not  result  in  catastrophic  mission  failure.  This  capability  can,  in  turn,  lead  to  a  greater  degree  of 
mission  flexibility,  another  advantage  of  UMVs,  as  mentioned  above. 

Operating  costs  are  also  often  cited  as  an  advantage  of  UAVs.  This  reduction  in  logistic  footprint  is  realized 
since  there  is  no  need  for  pilots  and  supporting  equipment.  For  example,  24  hour  surveillance  can  be 
maintained  by  a  single  U.S.  Army  tactical  UAV  Shadow  system,  which  includes  3  aircraft.  This  system  has  a 
logistics  footprint  of  about  16  soldiers  [9].  A  similar  capability  from  a  manned  aircraft  system  would  require  a 
much  larger  compliment. 

6.6. 1.3. 1.3  Workload 

Automation  is  often  implemented  to  reduce  operator  workload.  There  have  been  some  very  important 
examples  of  this  such  as  the  crew  reduction  from  three  to  two  in  commercial  aircraft.  The  1970s  saw  an 
increase  in  automation  which  reduced  workload  enough  on  commercial  flight  decks  to  lead  to  this  crew 
reduction.  However,  it  has  also  been  shown  that  automation  often  changes  the  nature  of  workload. 
The  automated  cockpit,  for  example,  changes  workload  from  physical  to  cognitive.  For  an  already  busy 
operator,  that  shift  can  dramatically  increase  their  overall  workload.  Attempts  to  lower  workload  through 
automation  need  to  be  carefully  evaluated  and  empirically  demonstrated. 

There  are  many  important  benefits  of  automation  as  discussed  above,  however  designers  must  take  great  care 
not  to  gain  through  automation  in  one  area  only  to  detract  from  another  area.  If  not  properly  addressed, 
the  “costs”  of  automation  can  be  greater  than  the  reward.  Some  potential  costs  of  automation  are  discussed 
below. 

6. 6. 1.3.2  Potential  Costs  of  Automation 

6.6. 1.3.2. 1  Mode  Awareness 

A  major  concern  regarding  human-automation  interaction  has  to  do  with  mode  awareness.  Too  many 
examples  are  available  that  show  crews  taking  an  action  that  would  be  correct  in  one  mode,  but  that  leads  to 
problems  in  the  present  mode  [10].  Consider  the  tragic  example  of  Korean  Airlines  007.  This  flight  from 
Anchorage,  Alaska  to  Seoul,  South  Korea  ended  in  a  tragedy  that  was  a  confluence  of  many  factors, 
but  the  initiating  cause  was  mode  awareness.  The  pilots  were  in  heading  hold  mode  which  keeps  the  aircraft 
on  the  generally  correct  heading  within  15  degrees.  This  is  adequate  for  short  distances,  but  as  will  be  seen  the 
error  grows  dangerously  large  over  longer  flights.  The  pilots  switched  the  mode  control  panel  from  heading 
hold  to  the  more  accurate  inertial  navigation  system  (INS).  However,  entry  into  INS  mode  requires 
satisfaction  of  two  conditions.  The  aircraft  needs  to  be  within  7.5  miles  of  the  route  and  pointed  in  the  general 
direction  of  the  route.  One  of  these  conditions  was  not  met.  The  aircraft,  therefore,  never  entered  INS  mode. 
Over  thousands  of  miles,  the  aircraft  drifted  200  miles  off  course  into  airspace  controlled  by  the  USSR  and  the 
event  ended  with  disastrous  consequences.  The  initial  cause  of  this  accident  was  the  pilots’  misunderstanding 
of  the  control  modes.  Clearer  enunciation  and  more  intuitive  control  of  the  modes  are  essential  to  making  the 
automation  more  useable  and  error  tolerant. 

6. 6. 1.3. 2. 2  Out  of  the  Loop 

In  a  highly  automated  system,  the  operator  may  be  asked  to  monitor  a  number  of  processes. 
If  the  automation  controlling  these  processes  is  largely  successful  and  failure  rates  are  low,  the  operator  may 
not  need  to  monitor  very  closely.  This  can  lead  to  the  operator  being  out  of  the  loop  when  a  failure  does  occur. 
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Recognizing  that  a  failure  has  occurred  and  getting  back  the  situation  awareness  needed  for  diagnosis  can  take 
a  critically  long  time.  This  concern  is  especially  prevalent  for  “semi-autonomous”  systems,  in  which  the 
system  is  highly  automated  without  a  human  in  the  loop,  until  something  goes  wrong.  The  role  of  the  human 
is  then  to  quickly  assess  the  situation  and  take  corrective  action,  but  having  been  out  of  the  loop,  this  task  is 
much  more  difficult.  Lee  and  Moray  called  this  “Out  Of  The  Loop  Un-Familiarity”  [11]. 

6.6. 1 .3.2.3  Knowledge  of  Automation  State 

Highly  related  to  being  out  of  the  loop  is  knowledge  of  automation  state.  In  order  for  the  operator  to  team  with 
the  automation,  he/she  needs  to  know  what  the  automation  is  doing  and  why.  The  automation  needs  to  be 
transparent  and  not  a  “black  box.”  The  lack  of  this  knowledge  will  lead  to  mode  awareness  problems, 
under  utilization,  high  workload  in  trying  to  determine  what  the  automation  is  doing  and  poor  situation 
awareness. 

6. 6. 1.3. 2.4  Over  Reliance 

Automation  bias  can  come  in  two  forms,  over  reliance  on  automation  and  under  reliance.  Over  reliance, 
also  called  complacency,  takes  place  when  operators  trust  the  automation  to  the  extent  that  they  no  longer 
cross  check  what  it  is  doing,  and  blindly  accept  its  direction.  An  example  of  complacency  was  discussed  by 
Azar  [12].  A  Panamanian  cruise  ship,  Royal  Majesty,  was  off  the  coast  of  Nantucket.  The  ship  was  being 
controlled  by  a  satellite  navigation  system,  which  failed.  Several  other  sources  of  navigation  information  were 
correct  and  available  to  the  crew.  These  however,  went  unmonitored  and  as  a  result  the  ship  ran  aground. 

6. 6. 1.3. 2. 5  Under  Reliance 

Automation  can  also  be  biased  toward  under  reliance.  High  false  alarm  rates  in  the  early  design  of  the  Ground 
Proximity  Warning  System  (GPWS)  led  pilots  to  disable  the  system  and  turn  off  the  automation.  This  has  also 
been  called  automation  disuse  by  Parasuraman  and  Riley  [13]. 

6.6. 1.3. 2. 6  Brittle  Automation 

Automation  is  brittle  when  it  works  only  in  specific  situations  and  doesn’t  generalize  well  to  unanticipated 
situations.  This  occurs  when  the  models  of  the  world  instantiated  in  the  automation  are  incomplete  or 
inaccurate.  This  can  cause  users  to  lose  confidence  in  the  automation  and  not  use  it  in  situations  where  it  may 
work  well  [14].  In  these  cases,  it’s  important  to  provide  a  cooperative  aid,  rather  than  a  fully  automated 
decision  or  solution.  In  this  way,  the  operator  can  provide  the  flexibility  needed. 

6.6. 1 .3 .2.7  Accountability 

An  interesting  aspect  of  higher  levels  of  automation  is  that  it  is  less  and  less  clear  who  is  accountable  for  an 
erroneous  action.  In  a  manned  system  the  pilot  or  driver  is  in  control  and  accountable  for  the  actions  of  the 
vehicle.  However,  especially  under  higher  levels  of  automation,  it  is  not  clear  that  the  UMV  operators  are 
making  the  decisions  that  account  for  actions  of  the  vehicle.  While  this  operator  is  still  legally  responsible, 
he  may  be  accepting  decisions  made  by  the  automation.  The  decision  logic  may  have  been  designed  by 
the  person  who  developed  the  automation  algorithms  or  the  software  engineer  that  produced  the  code. 
This  question  becomes  far  less  academic  when  UMVs  become  armed  vehicles. 
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6.6.1.4  Philosophies  and  Methodologies 

6.6. 1.4.1  Human-Centered  Automation 

The  introduction  of  glass  cockpits  and  subsequently  automation  to  piloted  fixed  wing  commercial  aircraft  led 
several  researchers  to  examine  the  transition  to  automated  cockpits  and  the  human  automation  interaction. 
Wiener  and  Curry  [15]  studied  the  transition  from  traditional  steam  gauge  cockpits  to  glass  cockpits. 
This  study  documented  problems  encountered  during  initial  transition  to  automation.  A  major  issue  noted  by 
Wiener  [16]  is  that  while  automation  may  reduce  small  errors,  it  may  invite  large  blunders,  such  as  the 
American  Airlines  accident  in  1996  [21].  Wiener  (1989)  [17]  found  that  the  present  generation  of  automation 
was  essentially  sound,  but  lacking  in  proper  user  interface  design. 

Billings  [18]  synthesized  much  of  the  extant  work  on  human  automation.  Billings  defines  human-centered 
automation  as  automation  designed  to  work  cooperatively  with  human  operators  in  the  pursuit  of  stated 
objectives.  Building  on  previous  work,  Billings  developed  guidelines  for  human-centered  automation,  shown  in 
Table  6-4. 


Table  6-4:  Human-Centered  Automation  Guidelines  [18] 

•  The  human  operator  must  be  in  command. 

•  To  command  effectively,  the  human  operator  must  be  involved. 

•  To  be  involved,  the  human  operator  must  be  informed. 

•  The  human  operator  must  be  able  to  monitor  the  automated  systems. 

•  Automated  systems  must  be  predictable. 

•  The  automated  systems  must  also  be  able  to  monitor  the  human  operator. 

•  Each  element  of  the  system  must  have  knowledge  of  the  others’  intent. 

•  Functions  should  be  automated  only  if  there  is  a  good  reason  for  doing  so. 

•  Automation  should  be  designed  to  be  simple  to  train,  to  learn,  and  to  operate. 


As  can  be  seen  from  earlier  examples,  many  of  these  guidelines  have  been  violated  in  interface  designs  that 
led  to  predictable  results.  Guidelines  can  be  useful  in  developing  an  underlying  philosophy  of  design. 
Following  them  to  specifically  design  an  interface,  however  can  be  very  difficult.  Other  approaches  have 
emerged  to  address  human-automation  interaction. 

6. 6. 1.4.2  Intelligent  Entities /Delegation 

Another  way  to  view  the  interaction  with  UMVs  characterizes  them  as  intelligent  entities  and  seeks  to 
delegate  authority  to  UMVs  in  a  systematic  way.  The  playbook  approach  [19]  uses  the  sports  analogy  that  the 
operator  is  the  coach  and  can  call  a  “play.”  The  intelligent  entities  know  what  their  roles  and  responsibilities 
are  within  that  play  and  can  execute  those  autonomously.  For  example,  an  operator  may  need  to  get 
information  about  a  particular  intersection  of  two  roads.  He/she  would  call  the  “recon  a  point”  play.  The  UMV 
asset  or  assets  know  what  this  means,  a)  move  into  a  clandestine  position,  b)  observe  the  intersection  for  a 
specified  amount  of  time  and  c)  return  the  sensor  information  to  the  operator.  These  plays  can  be  modified  by 
the  operator  as  he/she  sees  fit. 
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6.6. 1.4.3  UMVs  in  a  Network  Centric  System 

Yet  another  way  to  view  UMVs  is  as  information  sources.  A  commander  may  have  a  network  of  UMVs  at  his 
or  her  disposal.  Instead  of  specifically  commanding  any  asset,  he/she  would  instead  ask  the  network  for 
information.  For  example,  if  the  commander  needs  information  about  enemy  assets  in  a  certain  area,  he  can 
ask  the  network  for  that  information.  The  network  knows  what  assets  are  available,  and  which  are  needed  for 
this  request.  Based  on  the  priority  of  the  request,  the  network  assigns  assets  to  obtain  the  requested 
information  and  forward  it  back  to  the  requesting  commander.  In  this  network  centric  application,  the  UMVs 
are  efficiently  utilized  and  available  to  a  wider  tactical  community. 

These  approaches  are  not  mutually  exclusive  and  indeed  may  work  well  together.  Further,  additional  ways  to 
look  at  automation  have  also  been  suggested:  automation  as  an  associate  [20],  UMVs  working  as  a  team  or 
swarming,  and  automation  serving  as  an  electronic  crew-member  or  wingman.  The  task  of  the  interface 
designer  is  to  determine  how  these  may  be  adapted  to  best  serve  the  operator’s  needs.  They  may  serve  as  a 
toolkit  for  human-automation  interface  design. 


6.6.1.5  Interface  Implications  of  Automation 

This  section  summarizes  some  of  the  more  salient  points  of  the  preceding  discussions  and  draws  on  other 
sources  to  provide  issues  to  consider  in  UMV  interface  design.  They  are  not  meant  to  be  exhaustive  guidelines 
or  a  checklist,  but  rather  things  to  look  out  for  and  consider. 

1)  Automation  does  not  reduce  operator  workload  per  se;  it  may  change  the  nature  of  the  workload 
or  may  even  increase  it.  Automating  the  “stick  and  rudder”  flying  of  a  UAV  or  “hand  control”  of  a 
UMV  to  way  point  control  reduces  the  continuous  in  the  loop  manual  workload.  However, 
the  operator  is  now  a  supervisor  of  this  automated  system  and  has  to  monitor  the  vehicle  state  and  the 
automation  controlling  the  vehicle.  The  cognitive  workload  associated  with  this  supervisory  control 
may  well  be  higher  than  the  workload  of  physical  control. 

2)  It  is  critical  for  appropriate  use  of  automation  that  the  user  understand  how  the  automation  works  and 
what  mode  the  automation  is  in.  Without  this  understanding,  automation  bias  can  result. 

3)  There  is  an  inexorable  trade  off  between  higher  levels  of  automation  and  unpredictability.  As  systems 
move  more  and  more  toward  automation,  they  are,  by  definition,  less  predictable  and  therefore  don’t 
allow  predictability  or  insight  into  what  the  UMV  is  doing  (see  #  2). 

4)  All  automation  is  not  created  equal.  It  can  be  brittle,  unpredictable,  and  prone  to  bias.  Knowing  about 
these  pitfalls  is  half  the  battle.  A  designer  must  carefully  look  at  where  and  how  the  automation  may 
fail.  Is  the  mission  at  a  critical  point?  Does  the  automation  gracefully  degrade?  Does  the  operator 
know  the  automation  is  failing? 

5)  In  UMV  operation,  there  are  two  tasks:  vehicle  control  and  sensor  control/interpretation,  however, 
they  are  often  treated  as  one.  An  earlier  section  discussed  levels  of  automation,  the  first  taxonomy  is 
almost  entirely  in  terms  of  vehicle  control.  The  second  taxonomy  from  the  Army  Science  board 
addresses  a  bit  more  mission  oriented  tasks,  but  is  none  the  less  a  unitary  scale.  It’s  quite  likely  that 
the  sensor  interpretation  task  is  the  more  difficult  of  the  two  tasks,  and  the  more  difficult  to  automate. 
As  programmatic  goals  move  toward  control  of  multiple  UAVs  and  even  swarms,  it’s  important  to 
ask  who  is  interpreting  the  sensor  information  and/or  if  that  task  is  automated. 

6)  Beware  of  systems  described  as  “semi-autonomous.”  Many  current  systems  are  designed  such  that  a 
person  is  the  ultimate  monitor  and  failsafe,  but  if  automation  levels  are  high,  the  operator  is  prone  to 
being  out  of  the  loop  (see  above).  It  is  dangerous  to  design  systems  that  require  humans  to  manage 
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contingencies  when  the  automation  fails.  However,  it  should  be  noted  that  some  excellent  work  has 
been  done  in  this  area  under  names  of  shared  control,  traded  control  and  situation  adaptive  autonomy. 
The  reader  is  encouraged  to  fully  explore  these  areas  when  faced  with  “semi-autonomous”  systems. 

7)  UMVs  are  tools  to  provide  information.  It  should  be  remembered  that  UMVs  are  just  one  more  way 
to  provide  information  to  commanders  and  decision  makers.  As  such,  it  is  important  to  focus  on  the 
information  coming  in  and  the  information  presentation. 

8)  Lessons  learned  from  manned  aircraft  automation  and  other  domains  such  as  power  plants  offer 
invaluable  lessons  to  draw  from.  Years  of  automation  errors  and  poor  implementations  provide  rich 
information  sources. 

9)  Individual  differences  in  the  use  of  automation  make  predictions  difficult  (Parasurman  and  Riley 
[13]).  It  is  important  for  the  user  to  know  the  rationale  and  pay-offs  for  using  the  automation. 

There  is  no  formula  for  interfaces  with  automation.  There  are  guidelines  as  have  been  discussed,  but  most 
importantly  the  designer  needs  to  be  aware  of  these  issues,  communicate  to  the  user  and  continually  evaluate 
the  system. 

6.6.1.6  Research  Issues 

In  many  respects,  work  with  UMVs  and  human-automation  interfaces  is  in  its  infancy  and  many  research 
questions  remain.  A  great  deal  of  research  is  called  for  to  help  answer  these  questions.  In  the  area  of 
automation,  here  are  a  few  key  research  areas.  These  research  areas  are  also  addressed  in  Chapter  7  of  this 
volume. 

6. 6. 1. 6. 1  Level  of  Automation 

An  early  section  discusses  the  various  levels  of  automation  and  levels  of  control.  However,  it’s  not  clear 
which  of  these  or  what  combination  of  these  are  optimal.  Should  a  designer  always  try  for  the  highest  level  of 
automation  that  is  possible?  Increasing  levels  of  automation  will  lead  to  more  mission  capabilities,  but  at  what 
cost?  How  will  the  operator  manage  off-nominal  events? 

6.6. 1.6.2  Level  of  Involvement 

How  can  a  designer  increase  levels  of  automation,  but  keep  the  operator  informed  and  in  control?  Should  the 
designer  keep  the  automation  level  lower  to  keep  the  operator  in  the  loop?  Or  could  they  build  highly 
automated  systems,  but  build  in  tasks  that  keep  the  operators’  situation  awareness  high?  What  interfaces 
facilitate  this? 

6. 6. 1.6. 3  Automation  Transparency 

How  can  automation  interfaces  be  designed  so  that  the  actions  and  intentions  of  system  are  clear  to  the 
operator?  Pilots  often  struggle  with  understanding  what  the  flight  management  system  is  “trying  to  do.” 
This  needs  to  be  avoided.  The  intent  of  the  automation  needs  to  be  clear  to  the  operator  in  control.  How  this 
can  be  done,  to  what  extent  it  is  necessary,  and  what  interfaces  facilitate  this  are  major  research  questions. 

6. 6. 1.6. 4  Training 

The  question  of  operator  qualifications  have  become  very  important,  especially  when  in  control  of  UAVs. 
Do  the  operators  need  to  be  qualified  pilots  or  can  they  simply  be  trained  on  the  system  that  they  are  flying. 
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These  two  different  schools  of  thought  have  emerged,  but  there  is  little  research  available  addressing  this  issue. 
A  clear  definition  of  the  knowledge,  skills  and  abilities  to  operate  UMVs  is  needed  to  help  answer  this  question. 

There  also  remain  many  issues  not  raised  in  this  chapter,  but  that  will  need  to  be  addressed  by  UAV  designers 
in  the  near  future.  The  U.S.  Army  has  plans  to  use  manned  helicopter  and  UAVs  as  teams.  How  should  an 
interface  be  designed  that  optimises  teaming  between  manned  assets  and  unmanned  assets?  They  also  have 
plans  to  control  (at  some  level)  UAVs  from  helicopters.  What  is  the  optimal  level  of  automation?  Of  control? 
What  are  the  information  requirements? 

A  last  issue  that  will  be  raised  here  is  one  of  commonality,  or  lack  thereof.  Manned  aircraft  have  developed 
standards  over  the  years,  the  classic  example  being  the  T  arrangement  of  the  airspeed,  altitude  and  heading  in 
an  aircraft  cockpit.  This  has  allowed  pilots  to  move  from  one  aircraft  to  another  with  minimum  levels  of 
negative  transfer.  No  such  standards  exist  for  UMV  control  station  design.  This  has  led  to  vastly  different 
designs  by  each  manufacturer  and  the  result  that  operators  must  be  trained  very  specifically  on  each  platform 
control  station,  with  little  or  no  advantage  of  previous  learning.  This  lack  of  standard  design  must  be 
addressed  for  UMVs  to  reduce  training  costs,  logistics  and  operation  errors. 

6.6.1.7  References  for  Implications  of  Automation/Autonomy 

[1]  Oxford  Dictionary  of  Current  English  (1998).  Oxford  University  Press:  Oxford,  New  York. 

[2]  Sheridan,  T.B.  (2002).  Humans  and  Automation:  System  Design  and  Research  Issues.  John  Wiley  & 
Sons,  in  cooperation  with  HFES:  Santa  Monica,  CA. 

[3]  Sheridan,  T.B.  and  Verplank,  W.L.  (1978).  Human  and  Computer  Control  of  Undersea  Teleoperators 
(Man-Machine  Systems  Laboratory  Report).  Cambridge:  MIT. 

[4]  Parasuraman,  R.,  Sheridan,  T.B.  and  Wickens,  C.D.  (2000).  A  model  for  types  and  levels  of  human 
interaction  with  automation.  IEEE  Transactions  on  Systems,  Man  and  Cybernetics,  30(3),  286-297. 

[5]  Office  of  Secretary  of  Defense.  (2005).  Unmanned  Aerial  Vehicle  Roadmap:  2005-2030.  OSD: 
Washington,  D.C. 

[6]  Army  Science  Board.  (2004).  Autonomy  Levels  for  Unmanned  Systems  Workshop.  Huntsville,  AL. 

[7]  Standard  Interfaces  of  UAV  Control  System  (UCS)  for  NATO  UAV  Interoperability  (2004).  STANAG 
4586(2). 

[8]  http://www.aerovironment.com/area-aircraft/unmanned.html 

[9]  Dowell,  S.R.  (2005).  Shadow  Logistics  Footprint,  personnel  communications. 

[10]  Degani,  A.  (2003).  Taming  HAL:  Designing  interfaces  beyond  2001.  New  York:  Palgrave  MacMillian. 

[11]  Lee,  J.  and  Moray,  N.  (1992).  Trust,  control  strategies  and  allocation  of  function  in  human-machine 
systems.  Ergonomics,  35,  1243-1270. 

[12]  Azar,  B.  (1998).  Danger  of  automation:  It  makes  us  complacent.  APA  Monitor,  29(7). 

[13]  Parasuraman,  R.  and  Riley,  V.  (1997).  Humans  and  automation:  Use,  misuse,  disuse,  abuse.  Human 
Factors,  39(2). 
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[14]  Layton,  C.,  Smith,  P.  and  McCoy,  E.  (1994).  Design  of  a  cooperative  problem  solving  system  for 
enroute  flight  planning:  An  empirical  evaluation.  Human  Factors,  36(1),  94-119. 

[15]  Wiener,  E.L.  and  Curry,  R.E.  (1980).  Flight-deck  automation:  Promises  and  problems.  NASA  Technical 
Memorandum  81206.  Moffett  Field,  CA. 

[16]  Wiener,  E.L.  (1985).  Beyond  the  sterile  cockpit.  Human  Factors,  27,  75-90. 

[17]  Wiener,  E.L.  (1989).  Human  Factors  of  Advanced  Technology  (“Glass  Cockpit”)  Transport  Aircraft. 
NASA  Contract  report  NCC2-377. 

[18]  Billings,  C.E.  (1997).  Aviation  automation:  The  search  for  a  human-centered  approach.  Malwah,  NJ: 
Erlbaum. 

[19]  Miller,  C.A.,  Goldman,  R.P.,  Fubj,  H.B.,  Wu,  P.  and  Pate,  B.  (2004).  A  playbook  approach  to  variable 
autonomy  control:  Application  for  control  of  multiple,  heterogeneous  unmanned  air  vehicles.  American 
Helicopter  Society  60th  Annual  Forum  Proceedings,  pp.  2146-2157,  Alexandria,  VA. 

[20]  Miller,  C.A.  and  Riley,  V.  (1994).  Achieving  the  associate  relationship:  Lessons  learned  from  10  years 
of  research  and  design.  3rd  International  Conference  on  Human-Computer  Teamwork:  Cambridge,  U.K. 

[21]  Aeronautica  Civil  of  The  Republic  of  Columbia  (1996).  Aircraft  Accident  Report:  Controlled  Flight  Into 
Terrain,  American  Airlines  Flight  965,  Boeing  757-223,  N651AA,  Near  Cali,  Colombia,  December  20 
1995. 

6.6.2  Control/Display  Interfaces  for  Decision  Support 

Control  and  display  interface  design  has  always  been  a  key  concern  for  operator  station  development. 
However,  future  control  stations  that  employ  a  single  operator  to  control  multiple  UMVs  will  require  controls 
and  displays  that  not  only  enable  conventional  operator  tasking,  but  also  support  supervisory  tasks  as  well. 
These  tasks  include  quick  assessment,  judgment  and  reaction  to  the  appropriateness  of  the  automation’s 
changing  plans  and  actions,  as  well  as  continually  assessing  the  impact  of  these  changes  on  overall  mission 
objectives,  priorities,  etc.  Moreover,  as  the  number  of  controlled  vehicles  increase,  it  will  be  a  challenge  for 
the  operator  to  maintain  situation  awareness  through  long  periods  of  nominal  operations  interjected  with  short 
periods  of  time-sensitive  contingency  operations.  The  UMV  interfaces  need  to  effectively  cue  the  operator’s 
attention  to  critical  information  and  support  rapid  task  re-allocation.  Thus,  UMV  interfaces  must  be  tailored  to 
increasing  system  autonomy  and  novel  decision  support  systems. 

Research  is  needed  to  support  the  design  of  situation  assessment  and  decision  support  technologies  that 
maximize  flexible,  fault-tolerant  supervision  of  multiple  intelligent  semi-autonomous  UMVs  by  a  single 
operator.  This  technology  area  is  more  fully  discussed  elsewhere  in  this  report. 

Research  must  also  specifically  consider  the  design  of  tailored  controls  and  displays  for  UMV  decision 
support  systems.  Although  often  overlooked,  control/display  interfaces  are  critical  to  obtaining  maximum 
benefit  from  a  decision  support  system.  If  the  controls  and  displays  do  not  support  the  operator’s  decision 
making  process,  the  operator’s  effective  interaction  with  the  intelligent  automated  systems  will  be 
compromised  and  the  benefits  of  the  automation  will  not  be  fully  realized.  In  sum,  effective  decision  support 
control/display  interfaces  are  vital  to  the  UMV  operator’s  ability  to  make  timely,  accurate  decisions. 
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This  section  summarizes  a  survey  of  decision  support  literature  that  sought  to  determine  the  degree  to  which 
controls  and  displays  are  considered  in  the  design  of  previous  decision  support  systems,  identified  key 
decision  support  interface  types,  and  presented  a  notional  classification  system.  The  full  survey  is  more  fully 
described  elsewhere  [1]. 

6.6.2.1  Methodology  for  Decision  Support  Interface  (DSI)  Survey 

Thirty-four  DSI  research  papers  were  surveyed.  The  sampled  literature  included  journal  articles,  conference 
proceedings,  book  chapters,  and  technical  reports.  To  be  included  in  the  survey,  two  qualifications  had  to  be 
met.  First,  the  research  had  to  involve  a  problem  domain  in  which  a  human  user  was  challenged  by 
requirements  to  make  quick  and  accurate  decisions.  Thus,  the  documentation  did  not  need  to  address  the 
UMV  domain;  there  are  a  multitude  of  application  environments  that  require  decision  support,  the  findings  of 
which  are  potentially  applicable  to  UMV  supervisory  control.  Second,  the  research  had  to  involve  the 
evaluation  of  an  interface  concept  -  a  decision  support  control  and/or  display  was  manipulated  as  an 
independent  variable.  Thus,  this  survey  focused  on  empirical  evaluations  while  steering  away  from  papers  that 
broadly  addressed  decision  support  issues,  theory,  and  system  design. 

For  each  document  in  the  survey,  notes  were  made  in  multiple  columns  of  a  table  [1].  Besides  bibliographic 
information,  there  were  seven  columns  to  capture  information  on  the  nature  of  the  decision  support  interface 
used  in  the  research  and  the  results  of  the  evaluation.  The  following  further  describes  the  content  noted  in  each 
column: 

•  DSI  Category:  the  general  grouping  of  the  decision  support  interface  (DSI)  concept  (e.g.,  attentional 
cue,  status  display,  etc.). 

•  DSI  Description:  the  purpose  or  intended  benefits  of  the  decision  support  concept. 

•  Intelligent?:  the  computational  functionality  of  the  DSI  as  to  whether  it  is  based  on  artificial 
intelligence  algorithms,  knowledge-based  systems,  modeling  efforts,  or  simple  rule-based  logic. 

•  DSI  Control  Concepts:  identifies  any  control  concept  used  to  interface  to  the  decision  support. 

•  DSI  Display  Concepts:  describes  the  content/format  of  information  presented  in  a  DSI. 

•  Short  Summary:  an  overview  of  the  experimental  design,  independent  variables  and  user  tasks. 

•  Lessons  Learned:  summarizes  results  found  with  the  particular  decision  support  interface. 

•  Conclusions:  lists  DSI  findings  that  may  be  generalizable  to  other  decision  support  systems. 

6.6.2.2  Decision  Support  Interfaces:  Classification 

Entries  in  the  table  were  examined  to  derive  major  groupings.  First,  overall  grouping  of  the  decision  support 
interfaces  into  ‘controls’  and  ‘displays’  was  accomplished.  Next,  further  classification  was  accomplished 
based  on  other  factors,  such  as  the  degree  of  computational  functionality  (intelligence)  in  implementing  the 
interface  and  the  nature  of  the  support  it  afforded. 

6. 6. 2. 2. 1  Decision  Support  Controls 

Controls  were  first  classified  by  control  type  (conventional,  non-conventional).  The  majority  of  controls 
employed  in  the  decision  support  interface  literature  sampled  were  conventional  input  devices  -  mouse  and 
keyboard.  Less  often,  researchers  evaluated  decision  support  control  with  novel  devices.  These  included  voice 
recognition,  touch-screens,  and  reduced  keyboards  [2],  and  a  haptic  stick  [3].  Within  each  control  type, 
controls  were  grouped  by  technology  type. 
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It  was  also  deemed  useful  to  classify  by  the  method  used  to  achieve  control.  For  instance,  researchers  make 
the  distinction  between  controls  that  are  ‘bottom-up’  as  opposed  to  ‘top-down’  [4].  In  bottom-up  control, 
individual  elements  of  the  decision  problem  are  specified  in  detail,  and  the  user  then  receives  updates  on  each 
element.  In  top-down  control,  an  overview  of  the  decision  problem  is  formulated,  each  element  of  the  problem 
is  then  refined,  and  the  user  is  guided  down  through  the  hierarchy  of  information.  Another  sub-classification  is 
whether  the  control  employed  intelligent  filters  that  retrieve,  fuse,  and  manage  information  during  the  control 
process  [3].  In  contrast  to  having  intelligent  filters,  the  design  can  be  based  on  the  user  changing  system 
constraints,  parameters,  and/or  plans  which  results  in  the  initialization  of  processes  that  change  (i.e.,  control) 
system  states  [2].  Two  opposing  control  methods  make  up  a  final  defined  control  category:  closed  loop  control 
versus  open  loop  control.  With  open-loop  control,  no  feedback  is  sent  back  to  the  controller  because  the 
control  equation  is  well-accepted.  During  closed-loop  control,  feedback  is  provided  to  the  controller  so  that 
any  subsequent  input  brings  the  system  closer  to  the  goal  state. 

6. 6. 2. 2. 2  Decision  Support  Displays 

The  sampled  decision  support  interface  literature  described  many  types  of  displays.  These  were  grouped  into 
two  major  categories:  displays  that  are  simply  rule-based  and  ones  that  are  knowledge-based. 

Automated  warnings,  alerts,  and  cues  consisting  of  simple  notifications  (visual,  aural,  tactile)  when  a  rule  or 
goal  state  is  broken  (or  met)  constituted  one  common  type  of  rule-based  decision  support  interface  display. 
These  displays  can  be  multi-modal.  Attention  cuing  displays  aid  target  prosecution  and  highlight  critical 
information  or  points  of  interest  through  displays  technologies  such  as  synthetic  vision  symbology  overlaid 
conformal  on  a  camera  video  display,  automatic  target  cueing,  and  assisted  target  recognition  [5,6,7]. 
Status  displays  also  mainly  use  a  rule-base  to  present  the  decision  maker  with  key  parameters  and  their 
associated  values.  These  displays  can  provide  system  history  in  the  form  of  graphs,  intelligence  reports, 
or  logs.  Similarly,  status  displays  can  offer  classification,  filtering,  or  generalizing  of  situations  and  systems 
[8,9].  Action  recommendation  displays  take  status  display  information  and  use  another  rule-base  to 
recommend  a  course  of  action  to  the  decision  maker  (e.g.,  options,  strategies),  but  does  not  implement  the 
action  [10,11].  A  specific  example  is  a  Highway-in-the-Sky  display  that  provides  flight  guidance  [12]. 
Adaptive/adaptable  systems  are  another  type  of  rule-based  display.  Adaptable  systems  allow  the  operator  to 
initiate  automated  state  and  mode  changes,  while  adaptive  systems  allow  the  operator  or  automation  to  initiate 
the  changes  depending  on  the  automation  management  schema.  In  some  designs,  a  simple  rule-base  can  cause 
automated  mode  changes  when  the  operator  crosses  threshold  values  in  path  deviations,  EEG  index  (workload 
measure  based  on  electroencephalographic  signals),  or  altitude  (such  as  a  ground  collision  avoidance  system). 
Rules  could  also  trigger  the  system  automation  to  collect,  filter,  organize,  and  present  information  to  the 
operator  in  anticipation  of  upcoming  decisions. 

Knowledge-based  decision  support  displays  are  also  referred  to  as  intelligent  displays.  Definitions  of 
“intelligence”  vary,  but  many  include  the  ability  of  an  entity  to  achieve  goals  in  complex  real  world  situations. 
Some  definitions  specify  that  the  entity  must  perceive,  learn,  reason,  communicate,  and  act  [13]. 
Using  intelligent  systems  to  support  decision  making  is  most  appropriate  for  well-structured,  clearly  defined 
decision  situations.  Many  types  of  intelligent  displays  rely  on  intelligent  agents  that  work  in  the  background  to 
collect,  filter,  organize,  and  present  information  to  the  decision  maker.  With  an  intelligent  system,  status 
displays  can  also  provide  a  feed-forward  presentation  to  support  the  decision  maker  in  anticipating  future 
system  states  through  Gantt  charts  and  other  predictive  displays.  Intelligent  agents  can  also  expand  the 
capabilities  of  displays  in  adaptive  systems.  Other  types  of  knowledge-based  decision  support  displays  are 
plan  generators  and  evaluators.  With  these  displays,  operators  can  a)  generate  a  plan  and  then  have  the  system 
evaluate  (critique)  it;  b)  tweak  a  system-generated  plan  and  then  have  it  evaluated;  or  c)  use  the  intelligent 
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technology  to  predict  future  system  states  by  inputting  “what  if?”  scenarios.  The  generation  and  evaluation  of 
these  plans  are  usually  dependent  on  optimization  algorithms  or  knowledge-based  expert  system  modeling. 
Similarly,  intelligent  adaptive  decision  support  displays  rely  on  intelligent  agents  to  monitor  situations  and 
decide  what  information  to  present  to  the  user,  in  what  format,  and  at  what  time.  Some  intelligent  agents  may 
replace  some  of  the  operators  in  a  team  decision  making  environment.  To  make  effective  team  decisions, 
the  agent-user  collaboration,  negotiation,  and  dialogue  must  work  successfully  so  the  agent  can  provide 
aggregate,  inferential,  and  decision  information  to  the  operator(s)  without  information  overload. 


6.6.2.3  Decision  Support  Interface  Survey:  Lessons  Learned 

6.6.23.1  General  Findings 

One  objective  of  the  survey  was  to  determine  the  degree  to  which  controls  and  displays  are  considered  in  the 
design  of  decision  support  systems.  For  the  literature  sampled,  the  majority  dealt  with  display  interfaces  as 
opposed  to  control  interfaces.  This  finding  is  not  surprising  and  can  be  viewed  as  reflecting  the  larger  body  of 
research  dealing  with  displays,  compared  to  controls,  in  overall  human  factors  literature. 

Another  finding  raises  a  more  pressing  concern.  First,  note  that  all  the  sampled  literature  focused  on 
evaluating  a  control  and/or  display  interface  for  a  decision  support  system.  However,  less  than  a  third 
explained  the  rationale  for  the  control/display  concept  chosen  to  be  employed  with  the  particular  decision 
support  system.  In  other  words,  these  reports  didn’t  cite  published  research  or  a  previous  application  of  the 
control/display  that  supported  its  utility  in  the  decision  support  system  featured  in  the  report.  Although  this 
finding  may  only  reflect  incomplete  documentation  by  the  authors,  it  is  feared  that  it  is  indicative  of  an 
inadequate  prioritization  on  control/display  design.  Another  possibility  is  that  there  are  problems  inherent  in 
generalizing  existing  control/display  findings  to  decision  support  systems.  Regardless,  this  finding  supports 
the  need  for  additional  research  on  how  best  to  apply  control/display  interfaces  for  use  with  decision  support 
systems  for  UMV  operators. 

Below  is  a  summary  of  survey  findings.  Complete  results  can  be  found  in  [1]. 

6.6.23.2  Decision  Support  Controls 

The  literature  survey  indicated  that  the  use  of  non-conventional  controls,  such  as  speech  recognition, 
touch-screens,  and  reduced  (custom  function)  keyboards  generally  improved  operator  performance  and  either 
reduced  or  had  no  negative  effect  on  perceived  workload  [2].  Another  finding  was  that  in  many  decision 
support  systems,  there  are  alternate  control  methods  simultaneously  available  to  the  operator.  For  instance, 
the  decision  support  system  may  allow  the  operator  to  choose: 

•  Information  search/acquisition  method,  such  as  bottom-up  (start  with  detailed  query  and  get  updates/ 
notifications)  or  top-down  (global  perspective  with  guides  to  detailed  information); 

•  Information  assessment  method  (what  filters  or  constraints  are  employed); 

•  Whether  to  view  recommended  alternatives  or  only  the  “optimal”  solution; 

•  Control  input  device  (by  sketching,  typing,  or  speaking  novel  problem  solution  into  the  system). 

Empirical  results  suggest  that  these  control  choices  could  significantly  affect  the  decision  support 
effectiveness  and  overall  mission  performance.  For  instance,  choosing  to  select  a  path  recommended  by  the 
decision  support  system,  as  opposed  to  using  a  more  cumbersome  sketch/graphical  input  method,  can  bias  the 
operator’s  decision  [9]. 
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6.6.233  Decision  Support  Displays 

6. 6. 2. 3. 3.1  Rule-Based  Displays 

Automated  Warnings,  Alerts,  and  Cues 

Similar  to  decision  support  controls,  performance  with  non-conventional  decision  support  displays  was 
generally  better  than  with  conventional  visual  displays.  Multi-modal  displays  were  found  to  improve 
performance  for  many  tasks,  especially  when  used  as  a  redundant  cue  to  visual  information.  Examples  include 
haptic  [3],  auditory  [14],  and  voice  displays/feedback  [2,15].  Another  key  finding  related  to  cuing  displays  is 
the  potential  for  cognitive  tunneling.  Cognitive  tunneling  can  occur  when  the  operator  becomes  focused  on  the 
cue  being  displayed  to  such  an  extent  that  other  important  objects  in  the  view  are  not  attended  [16].  Research 
is  needed  to  explore  how  cue  displays  might  be  designed  to  minimize  the  negative  effects  of  cognitive 
tunneling.  Decision  support  cues  can  also  unintentionally  obscure  both  expected  and  unexpected  information 
[7].  Techniques  are  being  explored  to  minimize  this  problem  [17].  Finally,  there  is  a  possibility  that  operators 
will  over  rely  on  cues  displayed  from  the  decision  support  system,  resulting  in  operator  complacency  [15]. 

Status  and  Action  Recommendation  Displays 

In  general,  the  sampled  research  showed  that  status  and  action  recommendation  type  decision  support  displays 
were  beneficial  to  operators’  performance  in  the  task  environments  tested.  However,  the  reliability  of 
information  displayed  begins  to  have  a  more  consequential  impact  with  recommendation  displays.  This  is 
because  it  is  more  difficult  for  operators  to  judge  from  a  recommendation  display  that  underlying  data  is 
suspect.  Studies  have  shown  that  when  the  decision  support  system  has  reduced  reliability,  operator 
performance  is  better  with  a  status  display  by  itself  [10,18].  Additional  findings  suggest  displays  that 
incorporate  graphics  are  better  than  text-based  displays.  It  was  hypothesized  that  the  graphic  displays  were 
rated  higher  in  usability  because  they  support  human  pattern  recognition  or  recognition-primed  decision¬ 
making  [8]. 

6. 6. 2. 3. 3. 2  Knowledge-Based  (Intelligent)  Support 

Since  knowledge-based  displays  are  more  difficult  and  time-consuming  to  develop  and  require  a  well-defined 
domain,  empirical  testing  is  scarce  and  this  was  reflected  in  finding  only  a  few  relevant  documents  in  the 
literature  sampled.  One  evaluation  of  an  advisory  tool  designed  to  generate  flight  paths  around  weather 
showed  that  providing  a  display  to  aid  operators  to  manually  develop  their  own  plan  prior  to  getting  a  plan 
from  intelligent  agents  enabled  operators  to  catch  faulty  plans  better  compared  to  displays  that  only  provided 
an  auto-generated  plan  [19].  This  result  suggests  that  a  cooperative  process  between  the  operator  and  the 
decision  support  system  would  help  minimize  the  occurrence  of  automation  bias. 

The  nature  of  the  cooperative  process  can  also  impact  the  effectiveness  of  the  decision  support  system. 
One  study  [20]  demonstrated  that  operators  could  generate  plans  for  distribution  to  team  members  faster  with 
an  intelligent  planner  that  provided  displays  to  assist  the  operator  through  the  planning  process  (providing 
suggestions,  blanks  to  fill,  etc.),  than  with  an  intelligent  planner  that  critiqued  an  operator-generated  plan 
(displaying  how  good  it  was  according  to  critical  mission  measures).  This  illustrates  how  display  design  needs 
to  simultaneously  take  into  account  control  design  as  well.  In  other  words,  the  optimal  nature  of  the  operator- 
system  dialogue  for  capitalizing  on  the  benefits  of  the  decision  support  system  needs  to  first  be  determined 
and  then  supported  by  the  control/display  interface. 

Information  reliability  also  plays  a  role  in  the  effectiveness  of  decision  support  system  displays.  In  the  study 
mentioned  above  [20],  a  display  with  no  support  was  compared  to  three  different  types  of  decision  support 
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displays:  one  that  just  listed  raw  data,  one  with  the  data  classified  into  columns,  and  one  in  which  accuracy 
probability  information  was  also  provided.  By  manipulating  the  source  of  errors  (raw  data,  erroneous 
classification,  erroneous  probability),  the  results  suggest  that  presenting  the  data  classified  into  columns  was 
best,  despite  the  fact  that  subjective  ratings  indicated  that  operators  trusted  all  three  types  of  displays  equally. 
Also,  the  finding  that  operators’  trust  was  the  same  for  all  display  types  illustrates  the  value  of  presenting  key 
information  in  the  display  by  which  the  operator  can  judge  the  adequacy  of  the  data  and,  ultimately,  improve 
the  decision  making  process. 

As  an  example  of  a  promising  knowledge-based  decision  support  display,  consider  [21].  These  researchers 
developed  a  tactical  display  concept  consisting  of  a  set  of  PCE  (Predicted-Capability-Envelope)  contours 
overlaid  on  a  2D  map  representation  (Figure  6-31).  These  PCE  contours  describe  the  maneuvering  margins  of 
ownship  movements,  in  relation  to  other  moving  platforms’  motions  and  ownship  capabilities.  Capability 
prediction  is  a  model-based  concept  that  takes  dependencies  between  control  actions  and  effectiveness 
parameters  into  account.  This  involves  calculation  and  presentation  on  a  navigation  display  of  the  total 
maneuvering  margins,  the  so-called  PCE.  The  concept  was  developed  for  a  ship-maneuvering  environment. 
In  this  case,  the  predicted  margins  include  restrictions  due  to  boundaries  and  other  traffic  ships.  The  PCE 
represents  the  complete  reach  of  the  controlled  vessel  for  a  particular  time  horizon.  By  intersecting  the  PCE 
with  a  required  minimum  safety  distance,  an  integral  representation  of  (other  traffic)  threats  and  (controlled 
ship)  capabilities  is  obtained.  Provided  that  course  and  speed  of  other  ships  remain  the  same,  the  presented 
threats  will  be  geographically  stable  areas.  Navigators  may  consider  these  threats  as  obstacles  in  the  fairway. 
Thus,  an  integrated  navigation  display  is  obtained  which  provides  an  overview  of  the  ship’s  maneuvering  and 
collision  avoidance  information  for  a  particular  navigation  task.  With  respect  to  the  PCE  concept,  several 
in-depth  part-task  simulator  studies  were  performed  to  quantify  the  effects  of  using  PCE  on  navigator 
performance.  The  experimental  results  show  that  capability  prediction  is  far  more  effective  than  other 
prediction  information,  e.g.,  path  prediction  [22]. 
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Figure  6-31:  PCE  Display  Representation  as  an  Overlay  on  a  Radar  Display.  The  center  dot 
represents  ownship  position.  Starting  at  this  position,  an  area  is  marked  (thin  grey  envelope) 
for  which  proximity  information  is  calculated.  The  red  zones  represent  areas  in  which 
the  own  ship  will  be  within  one  mile  passing  distance  from  another  vessel. 
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6.6.2.4  Survey  Implications  on  UMV  Supervisory  Control 

Decision  support  interfaces  are  key  to  realizing  the  benefits  of  intelligent  automated  systems  that  facilitate 
single  operator  control  of  multiple  UMVs.  If  the  interface  fails  to  support  the  operator’s  decision  making 
process,  the  operator’s  ability  to  make  timely,  accurate  decisions  will  be  compromised  as  will  the  ability  to 
judge  the  operation  of  the  semi-autonomous  systems.  This  literature  survey,  however,  suggests  that  attention 
to  date  has  focused  on  development  of  the  knowledge-base  for  the  decision  support  rather  than  the  optimal 
design  of  the  controls  and  displays  used  with  the  decision  support  system.  Thus,  there  is  a  need  for  research 
that  systematically  addresses  interface  issues  for  various  categories  of  UMV  decision  support  systems. 

Some  directions  for  this  research  are  indicated  in  the  surveyed  literature.  For  instance,  research  on 
multi-modal  interfaces  (e.g.,  speech-based  input  and  visual/aural  redundant  displays)  supports  their  use  with 
decision  support  systems.  The  interaction  of  control  and  display  design  was  also  illustrated,  showing  that  the 
desired  operator/system  dialogue  or  collaboration  must  also  drive  interface  design.  Other  studies  demonstrated 
the  value  of  presenting  key  information  in  the  display  by  which  the  operator  can  judge  the  adequacy  of  the 
data  and,  ultimately,  improve  the  decision  making  process.  For  UMVs,  this  would  suggest  displaying  status 
information  for  each  UMV  under  supervision,  increasing  the  displayed  information  by  a  factor  of  the  number 
of  vehicles.  To  minimize  information  overload/retrieval  problems,  an  intelligent  system  which  highlights 
important  information  while  maintaining  the  availability  of  other  information  may  be  useful,  as  well  as 
innovative  display  approaches  that  support  anticipatory  decision-making. 

In  research  evaluating  candidate  UMV  controls  and  displays,  the  reliability,  bandwidth,  and  timeliness  of 
information  need  to  be  manipulated  to  determine  their  effect  on  the  utility  of  the  decision  support  interface. 
It  is  also  important  that  the  interface  design/evaluation  takes  into  account  issues  of  cognitive  tunneling, 
effective  information  retrieval,  and  automation  bias.  Workload  and  vigilance  demands  are  also  critical. 
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6.7  ADDITIONAL  CONSIDERATIONS  FOR  UMV  INTERFACES 

The  environment  and  methods  in  which  a  UMV  is  intended  to  operate  will  dictate  particular  UMV’s  control 
station  affordances  and  constraints.  This  section  examines  the  scale  and  variability  of  control  stations  for 
UMVs,  identifying  primary  affordances  and  unique  constraints  associated  with  each.  This  section  also 
discusses  issues  associated  with  controlling  UMVs  from  moving  platforms. 

6.7.1  Scale  of  UMV  Operator  Interface 

Scale,  in  this  context,  is  used  to  describe  the  constraints  imposed  upon  the  design  of  UMV  control  stations. 
The  scale  ranges  from  man  portable  to  large  space  platforms  and  all  are  limited  to  single  operator  control. 

6.7.1. 1  Man-Portable  Platforms 

In  man  portable  platforms,  the  UMV  is  ideally  very  small,  lightweight,  rugged,  and  easy  to  operate  [1]. 
It  is  essential  that  this  platform  meet  these  characteristics  because  a  typical  operator  will  be  transporting  the 
entire  UMV  system  along  with  other  mission  critical  equipment.  These  UMVs  are  usually  used  for  “what’s 
over  the  hill”  type  missions;  requiring  fairly  autonomous  operation  to  gain  time  critical,  nearby  information. 
The  control  stations  for  man  portable  UMVs  will  typically  be  no  larger  than  a  laptop  and  can  be  made  to  fit  on 
smaller  devices  such  as  a  PDA  or  head-mounted  displays.  For  example,  the  Pointer  UAV,  developed  by 
AeroVironment  Corporation,  is  operated  by  the  user  through  a  large  tablet-like  PDA  [2]  (Figure  6-32). 
Another  example  of  this  can  be  seen  in  [3].  However,  ruggedization  of  this  equipment  will  generally  increase 
weight  and  size. 


Figure  6-32:  Example  of  Man-Portable  Control  Station.  (Source:  http://www.aerovironment.com/). 
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6.7. 1.1.1  Affordances  Provided  by  Man-Portable  Platforms 

Of  the  three  main  classes  of  UMV  control  stations,  man  portable  stations  afford  the  fewest  resources  for 
interfacing  with  the  vehicle.  Often  times  man  portable  UMVs  require  direct  line-of-sight  control  of  the 
vehicle.  This  allows  more  immediate  responses  to  control  inputs.  The  short  duration  of  operations  and  lack  of 
control  station  equipment  should  minimize  the  amount  of  ergonomic  concerns  with  respect  to  body  posture, 
workstation  design,  etc.  Decreased  available  interface  real  estate  requires  simplified  control  inputs  into  the 
UMV  system,  and  varied  operating  environments  may  allow  for  more  unconventional  input  methods  into  the 
system  such  as  speech  recognition. 

6. 7. 1. 1.2  Constraints  Associated  with  Man-Portable  Platforms 

Interface  designs  must  be  optimized  for  essential  basic  functions,  leaving  little  room  for  displays  and  controls 
that  may  expand  capability.  Further,  the  operating  environment  of  a  man  portable  UMV  control  station  can 
vary  more  than  restricted  space  and  unlimited  space  platforms,  increasing  the  need  to  make  robust,  hardy 
equipment.  Control  input  devices  cannot  be  overly  sensitive  or  delicate  (e.g.,  a  PDA-style  stylus  may  be  too 
fragile  of  an  input  device  if  the  operating  environment  requires  all  weather  operations).  Display  screens  must 
be  able  to  be  viewed  under  less-than-ideal  conditions  (e.g.,  bright  desert  environments),  and  access  to 
Command  and  Control  (C2)  information  for  these  systems  is  limited.  Operators  must  act  as  the  vehicle 
director  and  perform  multiple  other  functions  that  need  to  be  executed,  requiring  a  large  amount  of  autonomy 
on  the  vehicle’s  part.  Man  portable  UMV  platforms  create  many  challenges  if  an  operator  needs  to  operate 
more  than  one  vehicle  at  a  time. 


6.7.1.2  Restricted-Space  Platforms 

Restricted  space  control  stations  are  characteristically  built  to  be  semi-mobile,  often  in  a  portable  trailer. 
However,  restricted-space  platforms  may  also  be  in  the  back  of  another  vehicle,  such  as  a  High-Mobility 
Multi-purpose  Wheeled  Vehicle  (HMMWV)  or  tight  quarters  on  a  ship.  Space  is  often  limited,  varying  based 
on  the  UMV  system  it  supports,  but  there  is  no  requirement  for  the  system  to  be  man  portable.  An  example  of 
a  restricted  space  control  station  can  be  seen  in  Figure  6-33. 


Figure  6-33:  Example  of  Restricted-Space  Platform. 
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ORGANIZATION 


6. 7. 1.2. 1  Affordances  Provided  by  Restricted-Space  Platforms 

UMV  restricted  space  control  stations  possess  much  more  interface  space  compared  to  man  portable  systems. 
Additionally,  the  UMV  control  station  operating  environment  can  be  managed  to  a  large  degree  with  respect 
to  the  station’s  temperature,  lighting,  noise,  etc.  This  allows  some  degree  of  flexibility  in  the  design  of  the 
displays  and  controls,  as  they  do  not  have  to  be  as  hardy.  The  increased  space  in  these  stations  also  allows 
operators  to  have  additional,  but  not  necessarily  vital  tools,  to  improve  areas  such  as  situation  awareness  and 
workload.  However,  it  is  important  to  remember  that  display  and  information  input  space,  while  larger  the 
man  portable  systems,  can  quickly  become  congested.  Another  benefit  to  the  restricted  space  control  station  is 
the  greatly  improved  access  to  C2  information  as  these  platforms  have  the  potential  for  SATCOM 
communications. 

6. 7. 1.2.2  Constraints  Associated  with  Restricted-Space  Platforms 

As  would  be  expected,  restricted  space  platforms  have  fewer  constraints  than  man  portable  stations,  but  more 
than  unlimited  space  designs.  Often,  achieving  a  good  ergonomic  layout  in  restricted  space  control  stations  is 
difficult.  Vehicles  operating  from  these  control  stations  usually  demand  high  levels  of  operator  oversight, 
requiring  multiple  display  and  control  surfaces.  Fitting  all  the  controls  and  displays  is  typically  the  first 
priority;  accommodating  the  human  is  secondary.  Maintenance  access  is  often  restricted  as  is  any  redundancy 
in  the  controls  or  displays.  Information  input  is  often  performed  through  the  use  of  conventional  mouse, 
keyboard,  and/or  joystick  devices.  Technologies  such  as  speech  recognition  can  ease  the  space  requirements 
for  control  surfaces  in  these  platforms  and  allow  for  more  display  area,  better  ergonomics,  and  increased 
capability  tools.  Human  interface  design  needs  to  consider  how  best  to  access  and  display  large  amounts  of 
information  on  few  displays  (i.e.,  what  is  an  appropriate  cognitive  distance  for  each  item,  how  much  digging 
for  information  should  be  required  -  information  depth  versus  breadth,  etc.). 


6.7.1.3  Large-Space  Platforms 

In  a  platform  where  space  is  abundant  and  more  than  able  to  accommodate  the  operators,  the  station  would 
most  likely  be  located  in  some  sort  of  bunker  or  building  [1].  These  arrangements  allow  for  the  greatest  degree 
of  control  with  respect  to  the  control  station  operating  environment  (e.g.,  temperature,  lighting,  noise,  etc.), 
but  the  least  amount  of  mobility.  These  systems  might  typically  be  large  platforms  with  highly  autonomous 
vehicles  performing  numerous  functions  that  are  all  tied  into  net  centric  feeds.  Some  potential  examples  of 
large-space  platforms  can  be  seen  in  research  with  data  walls,  as  shown  in  Figure  6-34. 


Figure  6-34:  Example  of  Large-Space  Platform  -  A  Data  Wall  at  U.S.  Air  Force  Research  Laboratory. 
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6. 7. 1.3.1  Affordances  Provided  by  Large-Space  Platforms 

Perhaps  the  most  appealing  aspect  of  a  large  space  platform  is  that  it  could  allow  for  a  truly  user-centered 
design  approach  as  defined  by  [4].  While  the  user-centered  design  approach  should  be  used  for  the  man 
portable,  restricted  space,  and  large  space  platforms,  it  can  more  fully  be  exploited  in  the  large  space  system. 
Using  this  approach,  the  system  could  be  designed  with  the  user  in  mind  at  all  stages,  maximizing  the 
technologies  that  would  allow  an  operator  to  function  at  peak  efficiency  and  effectiveness.  It  is  possible  to 
incorporate  all  expanded  capabilities  as  well  as  basic  functionality.  The  size  could  be  enlarged  to  fully 
accommodate  proper  ergonomic  design,  information  input,  and  display  space  needs  of  a  human  operator. 
In  the  large  space  platform,  it  is  possible  to  completely  control  the  operator’s  working  environment  -  ensuring 
no  stray  lights,  sounds,  or  other  disturbances  interfere.  This  type  of  platform  would  be  ideal  for  controlling 
multiple  UMVs  or  any  type  of  system  that  requires  crews  of  operators. 

6. 7. 1.3.2  Constraints  Associated  with  Large-Space  Platforms 

The  unlimited  space  design  is  not  without  constraints.  The  biggest  drawback  is  its  lack  of  mobility.  To  move 
the  entire  station  from  one  area  to  another  becomes  a  long  and  difficult  task.  There  is  also  an  issue  of 
oversaturation.  For  instance,  just  because  there  may  be  space  to  add  an  additional  monitor  to  the  workstation 
does  not  mean  it  would  improve  the  operator’s  performance.  Providing  an  operator  with  additional  channels 
of  information  to  monitor  can  decrease  overall  situation  awareness,  increase  workload,  increase  the  possibility 
of  errors  due  to  missed  information  or  misperceived  information,  increase  the  possibility  of  cognitive 
tunnelling,  as  well  as  a  host  of  other  problems.  Human  factors  engineering  would  need  to  ensure  the  system 
does  not  inundate  the  user  with  too  much  information  or  too  many  controls. 


6.7.1.4  References  for  Scale  of  UMV  Operator  Interface 

[1]  U.S.  Department  of  Defense.  Unmanned  Aircraft  Systems  Roadmap  2005.  Retrieved  November  30, 
2005,  from  http://www.acq.osd.mil/usd/Roadmap%20Final2.pdf 

[2]  AeroVironment,  Inc.  Pointer  FQM-151A:  Unmanned  Aerial  Vehicle  (UAV)  System.  Retrieved 
November  3,  2005,  from  http://www.aerovironment.com/area-aircraft/prod-serv/ptrdes.pdf 

[3]  Goodrich,  M.  and  Quigley,  M.  (2004).  Mini-UAV  telemetry  and  imaging  visualization  for  searching 
tasks  [Abstract].  Proceedings  of  the  Cognitive  Engineering  Research  Institute  Annual  Human  Factors  of 
UAVs  Workshop,  1.  Retrieved  December  1,  2005,  from  http://www.cerici.org/workshop/abstract/ 
Goodrich.pdf 

[4]  Norman,  D.A.  and  Draper,  S.W.  (Eds.).  (1986).  User  centered  system  design.  Hillsdale,  NJ:  Lawrence 
Erlbaum  Associates. 

6.7.2  UMV  Operation  from  Moving  Platforms 

An  interesting  and  even  more  complex  problem  arises  when  considering  the  control  of  UMVs  from  moving 
platforms.  One  possibility  of  this  is  a  man  portable  system  being  controlled  as  the  operator  is  on  the  run. 
The  operator,  now  the  moving  control  platform,  must  operate  a  UMV  while  actively  maneuvering  his/her  own 
body.  An  alternative  scenario  involves  a  UMV  operator  controlling  the  UMV  from  a  station  located  in  a 
moving  vehicle.  Operators  must  maintain  spatial  orientation  and  awareness  of  the  UMV,  while  simultaneously 
sensing  and  understanding  the  dynamics  of  their  own  vehicle  [1]. 
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The  lack  of  proprioceptive  feedback  inherent  with  UMV  control  can  increase  the  workload  for  operators 
trying  to  maintain  spatial  orientation  and  awareness.  Sensory  information  afforded  an  operator  is  drastically 
reduced  and  delivered  almost  entirely  through  the  visual  channel.  Operators  controlling  UMVs  from  moving 
platforms  receive  sensation  cues  completely  independent  (and  often  contradictory)  from  the  vehicle  being 
remotely  controlled.  This  problem  is  compounded  by  the  fact  that  the  operating  environments  of  the  vehicles 
may  be  different  and  could  change  during  the  duration  of  the  operator’s  control  of  the  UMV.  For  example, 
an  operator  controlling  a  UAV  must  maintain  spatial  orientation  and  awareness  of  the  UAV,  while  possibly 
operating  from  an  air,  ground,  sea,  or  underwater  vehicle.  The  proprioceptive  cues  of  the  ground,  sea, 
or  underwater  environments  can  vary  greatly  from  those  of  the  aerial  environment  and  would  compound  the 
mismatch  of  cues,  increasing  the  difficulty  of  controlling  a  UMV  and  potentially  leading  to  motion  sickness. 

In  the  above  scenarios,  the  operator  is  simply  operating  the  UMV  while  another  operator  controls  the  platform 
containing  the  UMV  control  station.  However,  it  is  also  plausible  that  a  single  operator  will  directly  control  a 
vehicle  (or  his/her  own  body  motion)  while  also  attempting  to  control  one  or  more  UMVs.  This  presents  some 
unique  challenges  [1].  While  it  may  reduce  manpower  and  equipment  needs,  it  may  also  require  some  control 
devices  and  display  surfaces  to  act  as  inputs  and  displays  for  both  the  manned  and  unmanned  vehicles.  Issues 
emerge  as  to  how  best  to  switch  the  control  and  display  between  the  vehicles,  and  which  input  devices  and 
display  surfaces  can  be  used  for  both  vehicles  and  which  should  be  solely  dedicated.  This  becomes  an  issue  of 
supervisory  control  -  to  what  degree  does  the  operator  need  direct  control  and  what  areas  of  operation  should 
be  made  autonomous? 

Regardless  of  who  is  controlling  the  operator’s  vehicle,  certain  key  concerns  have  been  identified  by  [2]  that 
must  be  addressed  for  UMV  operations  from  a  moving  platform: 

•  How  should  spatial  information  about  own-vehicle  and  controlled  vehicle(s)  be  displayed  to  the  UMV 
operator? 

•  Can  individual  differences  in  spatial  orientation  and  mental  rotation  be  used  as  criteria  for  selecting 
and  assigning  UMV  operators? 

•  What  types  of  training  and  simulation  systems  are  necessary  to  develop  the  necessary  skills  to 
manage  complex  multi-vehicle  dynamics? 

In  addition  to  the  challenges  of  navigation/spatial  orientation,  other  issues  of  concern  are  motion  sickness, 
biodynamic  interference  with  manual  control,  and  head-mounted  display  (HMD)  bounce.  Motion  sickness  can 
occur  when  the  visual  cues  of  motion  differ  from  those  perceived  by  the  inner  ear,  creating  a  sensory  conflict 
[3].  Motion  sickness  can  include  nausea,  dizziness,  disorientation,  and  increased  stomach  awareness.  Another 
issue  is  that  of  biodynamic  interference,  or,  the  input  to  manual  control  devices  from  the  shock  and  vibration 
of  rough  terrain,  transmitted  from  the  vehicle  to  the  operator  and  into  the  controls.  This  is  especially  common 
in  UGV  operations.  And  finally,  HMD  bounce  refers  to  the  phenomena  of  HMDs  resonating  at  certain 
frequencies  of  vertical  axis  vibration,  which  are  often  found  in  ground  vehicles  [4].  The  HMDs  “bounce” 
relative  to  the  face  and  eyes  of  an  operator,  degrading  visual  performance. 

Below  is  a  general  discussion  of  the  physical  ergonomic  aspects  of  mobile  (and  fixed)  UAV  control  stations. 
This  is  followed  by  a  discussion  of  a  few  of  the  specific  issues  surrounding  the  control  of  UAVs  from  an  air- 
based  control  station. 

6.7.2.1  Physical  Ergonomic  Aspects 

This  section  discusses  physical  ergonomic  aspects  to  be  considered  to  optimize  working  conditions  for  mobile 
and  fixed  UAV  workplaces  based  on  lessons  learned  in  the  design  and  testing  of  workplaces.  The  discussion 
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will  give  insight  in  important  issues  in  a  general  sense;  however,  detailed  solutions  will  not  be  given. 
These  details  can  be  found  in  applicable  handbooks,  literature,  reports,  and  such.  Most  of  the  issues  mentioned 
below  may  be  recognized  as  common  sense.  However,  recent  experience  has  shown  that  even  common  sense 
is  commonly  overlooked. 

The  physical  ergonomic  aspects  include:  anthropometry,  biomechanics,  climate  control,  workplace  lighting, 
vibrations/accelerations,  and  control  and  display  placement.  The  fixed  workplaces  are  defined  as  shelters/ 
containers  with  one  or  more  UAV  (control)  workplaces.  These  shelters/containers  are  usually  fixed  on  a 
certain  location.  However,  it  is  also  possible  that  these  workstations  will  be  operational  during  displacements; 
hence,  they  become  mobile  workplaces.  The  portable  workplaces  described  here  are  in  effect  soldiers 
equipped  with  devices  to  assess  information  coming  from  the  UAVs  or  even  to  control  the  UAVs. 

6. 7. 2. 1.1  Fixed  and  Mobile  Workstations 

6.7.2. 1.1.1  Anthropometry 

A  common  experience,  at  TNO  Defence,  Security  &  Safety,  is  that  ergonomic  specialists  are  asked  to  test  the 
anthropometries  of  a  workplace  once  the  design  is  more  or  less  frozen  without  any  prior  involvement  during 
the  already  fulfilled  design  process.  The  outcome  of  these  tests  is  usually  disappointing  because  the  physical 
human  operator  is  not  considered  properly:  the  operators  do  not  fit,  tall  operators  have  to  be  shoehomed  in  and 
small  operators  can  not  reach  applicable  controls  and  displays. 

Even  more,  the  specialist’s  remarks  may  be  such  that  an  easy  improvement  of  the  workplace  tested  is  not 
possible,  invoking  the  need  for  drastic  and  costly  design  changes  to  the  workplace.  The  most  common  cause 
for  all  this  is  that  anthropometries  have  not  been  considered  properly  during  design  activities  despite 
commonly  available  human  modelling  techniques  (see  Figure  6-35).  The  most  basic  approach  to  overcome 
this  issue  is  to  set  clear  and  proper  anthropometric  requirements  once  starting  a  new  project,  to  assess  these 
anthropometric  requirements  once  developed  and  finally,  to  test  the  actual  result  using  digital  modelling, 
prototypes  and/or  mock-ups. 


Figure  6-35:  An  Example  of  a  Digital  Human  Modelling  System: 
A  Manikin  in  a  CAD  Model  of  the  F16. 
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Today,  the  use  of  statistical  boundaries,  or  percentiles,  is  being  replaced  by  “cases”.  These  cases  define  the 
boundaries  based  on  manikins  that  have  to  be  accommodated.  The  advantage  of  cases  that  they  are  more 
accurate,  give  a  better  accommodation  result  and  they  are  easily  embedded  into  currently  used  human 
modeling  techniques. 

6.7.2. 1.1.2  Biomechanics 

Frequently,  the  following  question  is  put  forward:  what  is  the  amount  of  force  that  a  P5  female  can  exert? 
The  question  implies,  in  most  cases,  that  small  females  are  weak.  However,  it  must  be  noted  that  size/stature 
and  force  are  not  correlated  at  all!  There  are  small  females  that  can  exert  more  force  than  their  tall  male 
colleagues  and  vice  versa.  On  the  other  hand,  it  must  be  noted  that  the  reach  capabilities  of  smaller  persons  are 
more  limited,  possibly  affecting  the  amount  of  force  to  be  exerted. 

Biomechanics  come  into  play  for  UAV  control  stations  when  thinking  about  antennas  that  have  to  be  erected 
and  while  deploying  a  launch  bed  for  the  UAVs  themselves.  These  riggings  usually  require  a  lot  of  energy 
before  they  are  effectively  deployed.  The  amount  of  energy  and  forces  needed  must  be  in  accordance  with  the 
user  population  capabilities.  More  often,  the  situation  occurs  that  there  is  a  mismatch.  Again,  there  is  a  basic 
approach  to  prevent  mismatches  taking  place:  set  biomechanic  requirements,  assess  these  issues  when 
developing,  and  finally  qualify  the  design  in  field  tests  on  prototypes/mock-ups. 

6.7.2. 1.1.3  Climate  Control 

Climatic  requirements  are  usually  well  defined  and  tested.  For  instance,  an  entire  shelter  including  its 
equipment  is  tested  in  a  climatic  room  under  certain  conditions  (e.g.,  Al,  hot  dry  conditions,  STANAG  2895 
[8])  with  a  positive  result.  However,  in  the  actual  deployment,  under  Al  conditions  it  becomes  clear  that 
the  required  temperature  cannot  be  reached:  in  effect  it  is  too  hot.  The  mistake,  often  experienced  at 
TNO  Defence,  Security  &  Safety,  is  that  the  test  settings  do  not  represent  the  actual  working  conditions 
because  the  energy  dissipated  by  the  operators  (about  1.3  kW  per  person)  is  not  taken  into  account. 
The  solution  is  to  be  very  critical  for  climatic  test,  and  to  make  sure  that  the  testing  conditions  do  represent 
actual  working  conditions,  using  all  equipment,  using  all  heat  producing  elements,  etc. 

6.7.2. 1 . 1 .4  Lighting  and  Colours 

Displays  are  commonly  used  in  today’s  workplaces.  Blackout  lighting  (blue  or  red  light)  systems  are  also 
commonly  used  in  control  stations  to  cope  with  certain  tactical  conditions.  One  must  make  sure  that  the 
information  shown  on  the  displays  is  in  accordance  with  the  blackout  lighting  conditions.  Recent  experience, 
at  TNO  Defence,  Security  &  Safety,  showed  that  friendly  troops,  displayed  in  green,  were  not  recognizable 
under  red  light  conditions.  Other  elements,  like  enemy  front  lines,  even  disappeared  under  these  conditions. 
Another  problem  became  apparent  after  a  short  while:  certain  operators  (about  8  to  10%  of  all  males)  could 
not  see  certain  elements  at  all.  These  operators  were  unable  to  distinguish  certain  colors:  they  were  colour 
blind.  Therefore,  a  strong  directive:  take  tactical  lighting  conditions  and  colour  blindness  into  account  in  the 
design  of  displayed  information  in  order  to  prevent  any  problems. 

6.7.2. 1.1.5  Mobile  Workplaces 

Designing  workplaces  for  static  conditions  poses  enough  challenges  already  before  an  acceptable  compromise 
is  found  -  but  an  even  bigger  challenge  becomes  apparent  when  the  workplaces  have  to  be  used  during 
displacements  as  well.  Aspects  such  as  HIC  (head  injury  criteria)  and  legislation  for  safety  belts  come  into 
play.  This  section  will  not  discuss  all  the  rules,  but  will  discuss  an  elementary  issue  for  the  operator.  It  is  clear 
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that  safety  belts  are  elementary  for  mobile  workplaces.  The  actual  choice  for  a  retractable  or  fixed  safety  belt 
has  a  strong  influence  on  possibility  to  work  effectively.  Recent  experience,  at  TNO  Defence,  Security  & 
Safety,  confronted  operators  with  fixed  belts  disabling  them  to  reach  the  controls  and  to  read  any  display 
information:  they  were  fixed  to  their  seats  and  could  not  reach  any  workplace  element.  Therefore,  a  strong 
word  of  advice:  opt  at  first  for  a  retractable  safety  belt  and  make  sure  that  the  operator  is  not  hampered  while 
working  due  to  any  safety  system. 

6.7.2. 1.1.6  Nuclear,  Biological  and  Chemical  (NBC)  Conditions 

It  is  a  fact  that  our  troops  are  confronted  with  NBC  conditions.  Even  under  these  circumstances  one  must  be 
able,  when  wearing  protective  gear,  to  effectively  operate  UAVs.  It  is  clear  that  protective  gear,  especially  the 
NBC  gloves,  affect  dexterity.  And  the  amount  of  energy  that  can  be  exerted  is  affected  as  well  due  to  the 
isolating  properties  of  the  suit  and  the  limited  breathing  possibilities  when  wearing  a  mask.  All  these  elements 
combined  will  have  to  be  taken  into  account  for  all  operations  inside  and  outside  (e.g.,  when  rigging  antenna 
arrays,  setting  the  UAV  launch  station)  the  UAV  control  station. 

6. 7. 2. 1.2  Portable  UA  V  Workstations 

There  seems  to  be  an  apparent  need  for  field  operators  (e.g.,  soldiers,  policemen,  firefighters),  to  have  a 
mobile  office.  In  some  cases,  the  operator  is  simply  equipped  with  an  advanced  mobile  telephone.  In  other 
cases  they  have  to  carry  a  complete  laptop  while  working.  The  result  is,  in  most  cases,  not  fully  satisfactory: 
the  equipment  does  not  perform  properly  (e.g.,  one  cannot  access  the  worldwide  web  properly  using  a  mobile 
phone  due  to  the  limited  size  of  the  display  and  input  media)  or  it  is  simply  too  bulky  (e.g.,  a  complete  laptop, 
including  power  source,  is  not  compatible  with  standard  battlefield  operations).  Recently,  a  system  was 
developed  as  a  prototype  system,  by  TNO  Defence,  Security  &  Safety,  with  a  helmet  integrated  vision  device 
and  a  small  touchpad  as  input  device,  in  the  Soldier  Modernization  Program  (see  Figure  6-36).  This  system 
basically  focuses  on  the  needs  for  the  operator  in  the  battlefield:  some  operators  only  need  a  display  for  a  short 
period  of  time  (e.g.,  a  soldier  needing  information  concerning  his/her  whereabouts),  other  operators  need  more 
information  (e.g.,  a  group  commander  needs  more  information  and  has  an  additional  need  to  input  data). 


Figure  6-36:  A  Soldier  Equipped  with  Various  Portable  Systems  for  Advanced  Field  Operations. 
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The  basic  thought  behind  this  system  is  that  it  is  fine-tuned  to  the  operator’s  needs.  This  system  is  not  yet  used 
or  tested  in  operational  circumstances;  this  will  take  place  in  the  beginning  of  2006.  An  elementary  issue  to  be 
focused  on  is  the  effect  of  the  new  system  on  battlefield  operations.  It  is  already  apparent  that  some  users  will 
have  to  change  their  standard  operational  procedures  because  they  will  have  an  additional  task  to  perform: 
the  task  of  information  management. 


6.12.2  Manned-Unmanned  UAV  Teaming  from  Air-Based  Control  Station 

Most  currently  used  UAVs  are  controlled  from  a  (closed)  container  Ground  Control  Station  (GCS).  The  GCS 
operators  are  generally  concerned  with  only  one  UAV  system.  Here,  the  Mission  Commander  communicates 
the  acquired  intelligence  to  other  units.  In  past  years,  experimental  setups  have  been  developed  in  which  the 
information  from  the  UAV  sensors  is  more  directly  presented  to  aircraft  during  a  mission.  A  pioneer  project 
was  U.S.  Army’s  AMUST  (Airborne  Manned  Unmanned  System  Technology  [5]).  Here,  an  Apache 
helicopter  and  a  Hunter  UAV  form  a  team,  where  the  UAV  may  perform  all  kinds  of  useful  sidekick  tasks, 
such  as  reconnaissance,  laser-designation  of  targets  and  acting  as  decoy.  The  UAV  can  be  either  controlled 
from  a  GCS  or  by  the  Apache’s  co-pilot/gunner. 

The  AMUST  concept  adds  to  the  complexity  of  maintaining  situation  awareness:  the  operator  (the  co-pilot/ 
gunner)  has  to  deal  with  at  least  three  spatial  frames  of  reference,  namely  the  world,  the  helicopter  and  the 
UAV.  Our  first  research  questions  were:  Is  such  a  co-pilot  able  to  build  up  situation  awareness  involving 
multiple  platforms,  and  how  is  this  situation  awareness  affected  by  being  on  board  one  of  the  platforms? 
These  questions  were  investigated  in  the  first  study  described  below. 

In  the  AMUST  concept,  the  co-pilot  interacts  with  the  UAV.  In  future  operation  concepts,  e.g.,  Network 
Enabling  Capability,  ‘sensors’  and  ‘shooters’  are  allocated  more  dynamically.  In  a  second  study,  briefly 
described  here  as  well,  we  investigated  the  possible  benefits  for  the  pilot  in  having  available  a  UAV  image  in 
the  cockpit  when  performing  a  simulated  Close  Air  Support  mission. 

6. 7.2.2. 1  Multi-Platform  Situation  Awareness 

De  Vries  and  Jansen  [6]  conducted  a  simulator  experiment  to  assess  the  situation  awareness  of  a  co-pilot  who 
controls  a  UAV  platform  while  on-board  a  moving  platform,  in  this  case  a  helicopter.  The  co-pilot  has  to 
integrate  the  reference  frame  of  the  UAV  with  that  of  his  own  platform.  Participants  serving  as  UAV 
operators  were  seated  behind  a  UAV  console  situated  in  a  helicopter  mock-up.  The  console  displayed  an 
electronic  map  of  the  geographical  situation  of  a  UAV,  helicopter  and  a  few  other  objects.  For  each  simulator 
trial,  the  participants  watched  the  movements  on  the  electronic  map  display  for  a  few  minutes  after  which 
their  situation  awareness  was  assessed  using  an  electronic  questionnaire.  Questions  could  be  related  to  an 
earth-fixed  coordinate  system  (compass)  or  be  orientation-based  (relative).  Questions  addressed  the  helicopter, 
UAV,  or  a  formation  of  tanks.  The  main  dependent  measurement  was  angular  error,  i.e.,  the  difference 
between  the  direction  indicated  by  the  participant  and  the  real  direction.  The  main  experimental  manipulation 
was  a  representation  of  the  outside  scenery  on  a  180-degree  cylinder-shaped  screen.  Indicators  were  presented 
to  show  the  movements  of  the  helicopter.  The  simulation  was  presented  in  two  work  environments  for  the 
UAV  operator:  in  a  ground  control  station  setting  and  on-board  a  moving  platform  (helicopter). 

The  most  important  results  from  this  study  are  presented  in  Figure  6-37.  The  Figure  depicts  for  the  six 
different  question  types  the  angular  error  in  indicating  the  location  of  an  object.  For  most  of  the  question 
types,  the  right  bars  (corresponding  with  the  condition  of  the  UAV  operator  aboard  the  helicopter)  are  much 
higher  than  the  left  bars  (corresponding  with  the  situation  of  an  operator  in  a  ground  control  station). 


6-94 


RTO-TR-HFM-078 


dMNATO 
WP  I  OTAN 


ADVANCED  UMV  OPERATOR  INTERFACES 


This  indicates  that  in  general  terms,  performance  is  worse  in  conditions  in  which  we  simulated  that  the  UAV 
operator  is  aboard  one  of  the  moving  platforms  (except  when  questions  were  asked  with  respect  to  the 
platform  itself).  Note  that  the  task  itself  is  not  different  for  the  stationary  and  moving  operator:  just  look  at  the 
electronic  map  and  build  up  a  spatial  awareness  as  good  as  possible.  The  results  revealed  that  it  is  indeed  more 
problematic  to  maintain  a  multi-platform  situation  awareness  while  being  aboard  a  moving  platform. 
Apparently,  when  the  operator  is  in  a  situation  where  one  spatial  frame  of  reference  is  dominant  (i.e.,  while 
aboard  a  helicopter),  it  is  very  hard  to  process  spatial  information  from  other  perspectives. 


Figure  6-37:  Angular  Error  in  Indicating  an  Object  Location  for  the  Six  Question  Types. 
The  left  (blue)  bars  refer  to  the  situation  of  an  operator  in  a  ground  control  station; 
the  right  (red)  bars  to  the  situation  of  an  operator  aboard  a  simulated  aircraft. 


6. 7. 2. 2. 2  Using  Real-Time  UA  V Images  while  Conducting  a  Close  Air  Support  Mission 

The  above  research  has  shown  that  performance  drops  when  multiple  reference  frames  need  to  be  integrated 
while  performing  a  spatial  situation  awareness  task.  In  a  second  experiment  [7],  a  more  critical  situation  was 
investigated  in  which  misinterpretation  of  spatial  information  directly  resulted  in  failure  of  a  simulated  Close 
Air  Support  mission.  A  situation  was  investigated  in  which  a  pilot  used  a  UAV  image,  presented  in  the 
cockpit.  As  the  UAV  camera  has  a  viewpoint  that  differs  from  that  of  the  pilot,  the  pilot’s  interpretation  of  the 
spatial  layout  of  the  scenery  may  be  prone  to  error  (e.g.,  if  the  UAV  flies  in  the  opposite  direction,  the  object 
on  the  left  in  the  sensor  image  is  actually  on  the  right  in  the  pilot’s  perspective).  The  aim  of  the  research  was 
to  minimize  the  chance  of  such  errors  by  rotating  the  UAV  sensor  image  such  that  its  orientation  is  always 
aligned  with  the  pilot’s  spatial  frame  of  reference.  In  this  study,  four  military  pilots  performed  several  Close 
Air  Support  missions  with  six  different  display  configurations.  The  orientation  of  the  electronic  map  was 
either  North-Up  or  Heading-Up.  The  UAV  sensor  image  was  either  absent,  present,  but  non-aligned 
(i.e.,  unadjusted  image  orientation:  the  image  is  presented  as  seen  from  the  UAV  viewpoint),  or  present  and 
aligned  with  the  orientation  of  the  electronic  map  (adjusted  image  orientation:  the  image  was  presented  as  if 
the  sensor  was  placed  on  the  helicopter). 

The  pilots  reported  that  they  generally  preferred  the  Aligned  UAV  sensor  image  in  combination  with  a 
Heading-Up  map.  This  preference  was  reflected  in  their  performance,  depicted  in  Figure  6-38:  targets  were 
identified  twice  as  fast.  Strikingly,  flying  performance  was  also  better  when  an  aligned  UAV  image  was  used. 
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Figure  6-38:  Time  to  Target  Identifications  for  the  Six  Display  Configurations. 


6. 7. 2. 2. 3  Conclusions 

In  comparing  the  (simulated)  situations  of  a  stationary  UAV  operator  and  a  moving  UAV  operator  it  appeared 
that  it  is  very  hard  to  process  spatial  information  from  multiple  perspectives  simultaneously  when  one 
perspective  is  dominant  (here,  when  being  aboard  a  helicopter).  In  experiments  on  presenting  UAV  images  in 
the  cockpit  while  performing  a  Close  Air  Support  mission  it  was  investigated  whether  the  interpretation  of 
(the  non-dominant)  UAV  image  information  was  facilitated  by  aligning  that  perspective  with  the  dominant 
perspective  of  the  helicopter.  Based  on  the  results  and  pilot’s  reports  it  was  concluded  that  the  availability  of  a 
UAV  sensor  image  in  the  cockpit  of  an  aircraft  only  improves  mission  performance  when  its  orientation  is 
aligned  with  the  aircraft. 
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6.7.3  Other  Issues:  Latency,  Trust,  Bandwidth 

As  well  as  the  factors  mentioned  above,  there  are  also  a  number  of  other  considerations  that  have  human 
factors  implications  for  a  UMV  system.  This  section  of  the  report  summarises  a  few  of  the  factors  that  are 
more  likely  to  be  encountered. 

6.7.3. 1  Latency 

In  this  instance,  latency  refers  to  the  time  delay  between  the  UMV  operator  making  a  control  input  and  the 
feedback  received  by  the  operator  from  the  system  to  confirm  that  the  control  input  has  had  an  effect. 
Such  latencies  are  particularly  noticeable,  in  the  UMV  context,  when  pointing  sensors  and  issuing  commands. 

Early  research  suggested  that  as  latencies  increased,  operator  performance  decreased  in  a  linear  fashion  [3], 
however  later  research  showed  this  to  be  oversimplifying  the  issue.  For  example,  Asbery  [1]  varied  the 
latency  in  feedback  between  28  ms  and  2000  ms  for  a  target  tracking  task,  and  found  that  there  was  a  sharp 
initial  decrease  in  performance  with  latencies  up  to  100  ms  with  the  detriment  on  performance  diminishing 
thereafter  until  1000  ms  where  it  levelled  out  at  a  consistently  poor  level.  This  suggests  that  the  claim  of  a 
linear  relationship  to  be  inaccurate.  Asbery  also  found  that  the  operators  employed  strategies  to  cope  with  a 
high  degree  of  delay  between  action  and  response.  When  latencies  increased,  the  participants  seemed  to  lose 
sense  of  the  orientation  of  the  camera  system.  To  overcome  this,  some  participants  delayed  their  inputs  until 
the  camera  ‘caught  up’  with  their  previous  action.  In  general,  trajectories  became  oscillatory  in  nature  because 
of  the  overshoot  generated  by  the  high  latency. 

It  has  also  been  shown  that  latency  may  have  very  precise  ‘effect  boundaries’  where  a  certain  range  of 
latencies  will  have  the  largest  effect  on  performance  at  a  task.  For  example,  studies  have  shown  that  for  a 
simple  control  task  (e.g.,  man-in-the  loop  UMV  control)  there  is  no  noticeable  effect  of  latency  between  0  and 
500  ms,  but  a  sharp  decrease  in  operator  performance  between  500  ms  and  1000  ms.  Similar  patterns  of 
results  have  been  found  in  a  trials  investigating  pilot’s  landing  ability  using  synthetic  visual  aids.  In  general, 
the  findings  from  research  in  this  area  support  the  view  that  the  impact  a  particular  level  of  latency  will  have 
on  a  task  is  linked  to  the  degree  of  control  required  to  complete  the  task.  A  relatively  low  latency  will  have  a 
much  more  pronounced  effect  on  a  task  involving  the  operator  making  many  minor  adjustments  than  a  task 
involving  only  general  large-scale  control  movements. 

Research  on  latency  in  virtual  reality  systems  [2]  found  that  latencies  in  update  following  a  head  movement 
had  a  more  profound  effect  on  the  visual  system  than  latencies  following  hand  movement.  It  was  suggested  by 
the  researchers  that  because  hand  movement  is  a  feed-forward  process,  there  is  relatively  little  effect  due  to 
latency,  as  the  control  is  not  based  on  feedback  from  the  visual  system.  On  tasks  that  require  visual  feedback 
to  monitor  the  results  of  actions,  latencies  have  a  much  more  noticeable  effect. 

To  conclude,  latencies  do  not  correlate  linearly  with  performance.  They  have  specific  ‘effect  boundaries’  that 
will  depend  on  the  type  of  task  undertaken.  Latencies  may  have  a  less  pronounced  effect  if  the  task  requires 
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only  feed- forward  control  and  not  visual  feedback.  Tasks  requiring  fine  motor  control  are  in  general  more 
severely  impacted  by  shorter  latency  periods.  Specific  guidance  on  the  likely  acceptable  level  of  latency  for  a 
given  type  of  task  is  given  in  various  standards.  For  example,  UK  Defence  Standard  00-25  [4]  lists  the 
following  recommendations  for  acceptable  response  times  associated  with  human-computer  interaction: 


•  <50  ms 

•  100  ms 

•  200  ms 

•  1-2  s 


From  movement  of  pointing  device  to  movement  of  cursor  or  marker. 

Pressing  a  key  to  character  being  displayed. 

Selection  of  displayed  object  (field,  button,  menu  option)  to  object  appearing  as  selected; 
Selection  of  menu  header  to  menu  being  displayed;  Selection  of  scroll  button  to 
completion  of  scroll  of  one  line  of  text. 

Completion  of  user  input  to  display  of  error  indication. 


•  2  s  Request  for  next  page  of  information  to  completion  of  one  page  change;  Completion  of 

user  input  to  completion  of  simple  process;  Completion  of  display  manipulation  request  to 
completion  of  display  change  (e.g.,  open  a  window;  zoom). 


6.7.3.2  Trust 

The  UMV  operator  is  required  to  make  accurate  and  timely  decisions.  As  they  are  physically  removed  from 
the  vehicle  they  are  controlling,  they  are  reliant  on  the  information  provided  to  them  from  external  sources, 
most  likely  from  the  UMV  itself,  and  their  experience  of  similar  situations  to  make  these  decisions. 


Trust  relates  to  the  operators’  willingness  to  rely  on  the  information  presented.  Clearly  the  operators  must  trust 
that  the  information  provided  is  accurate  if  they  are  to  use  it  effectively  in  the  decision  making  process, 
rather  than  trying  to  ‘second-guess’  the  situation  and  relying  solely  on  previous  experience.  In  addition, 
in  UMV  systems  that  rely  on  the  operator  acting  as  a  ‘system  supervisor’,  the  operator  must  trust  that  the 
automation  in  the  system  is  carrying  out  the  tasks  associated  to  it  in  order  for  this  mode  of  operation  to  work 
effectively.  Without  this  level  of  trust,  the  operators’  workload  would  increase  significantly  as  they  attempt  to 
monitor  the  system  too  closely. 


A  discussion  on  decision  making  is  outside  the  scope  of  this  report,  although  the  reader  is  encouraged  to  read 
one  of  the  many  texts  available  on  human  decision  making.  Instead  this  report  will  focus  on  issues  and 
research  directly  relating  to  trust. 


There  are  a  number  of  theories  of  trust.  For  example,  Muir  [8]  noted  that  the  study  of  trust  was  under 
researched  in  the  engineering  psychology  literature.  She  suggested  that  while  the  research  may  be  lacking  for 
trust  in  Human  Machine  Interaction  (HMI),  it  would  be  possible  to  extend  existing  theories  of  interpersonal 
trust.  Muir  built  on  the  theories  of  Rempel,  Holmes  and  Zanna  [9]  to  propose  a  three  factor  theory  of  how  trust 
develops.  Early  on  in  the  operator-machine  relationship,  the  predictability  of  the  system’s  decision  is 
important  in  building  trust.  A  system  that  makes  unpredictable  decisions  will  not  be  trusted.  Once  the  system 
is  perceived  as  predictable,  an  operator  will  trust  a  system  more  if  it  is  deemed  dependable .  Dependability 
evolves  over  time  as  the  system  operator  makes  generalisations  about  the  specific  actions  of  the  system  to  a 
broader  set  of  attributes  of  the  system.  Dependability  will  be  particularly  strong  if  the  machine  is  able  to 
perform  accurately  under  risky  situations.  The  final  stage  involved  the  development  of faith,  the  belief  that  the 
system  will  be  dependable  in  the  future.  During  this  stage  of  the  ‘relationship’  between  the  operator  and  the 
UMV,  transparency  in  the  UMVs  decisions  will  be  important  in  maintaining  trust. 
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This,  and  other  models  of  trust  applicable  to  UMV  systems,  tends  to  neglect  the  characteristics  of  the  operator 
in  favor  of  aspects  of  the  system.  For  example  Kelley  et  al.’s  model  [10]  does  take  the  operator  into  account  to 
a  limited  degree.  In  this  model,  trust  is  gained  and  influenced  by  the  competence  of  the  system  (as  summarised 
by  earlier  models,  described  above),  understanding  and  self-confidence.  Self-confidence  contains  factors  such 
as  skills,  experience  and  faith  which  directly  relate  to  the  operator  of  the  system.  Similarly  understanding 
contains  factors  that  arise  as  a  result  of  the  operator  and  UMV  in  combination;  predictability  and  familiarity 
for  example.  This  model  was  originally  constructed  to  understand  trust  within  the  Air  Traffic  Management 
(ATM)  environment,  although  it  can  be  applied  to  many  areas  of  human  machine  interface.  The  model  also 
examines  trust  at  a  broader  level,  fitting  it  into  a  model  of  system  acceptability  alongside  factors  of  teamwork 
and  situation  awareness  (Figure  6-39). 


Figure  6-39:  Trust  in  ATM  Context  (Kelley  et  al.,  [10]). 

Most  models  are  lacking,  as  they  generally  ignore  the  properties  of  the  operator.  Others  include  the  operator, 
but  only  when  factors  such  as  confidence  and  training  have  been  explored  [5].  Additional  attributes  of  the 
operator  may  also  have  an  impact  such  as  individual  differences  in  overall  propensity  to  trust. 

6.7.3.3  Bandwidth  and  Update  Rate 

The  bandwidth  of  the  communication  link  between  the  operator  and  the  UMV  and  the  update  rate  of  the 
information  that  is  provided  are  important  considerations  in  UMV  system  design.  In  order  to  control  a  system 
effectively,  the  operator  requires  the  right  amount  of  information  to  be  provided  at  the  right  time. 

Bandwidth  limits  the  size  of  the  information  passed  down  the  communications  link,  whilst  update  rate  refers 
to  how  often  the  information  is  refreshed.  Both  are  interlinked;  low  bandwidth  links  may  impose  a  limit  on  the 
speed  that  information  can  be  transmitted.  Bandwidth  may  be  restricted  by  technological  limitations  or  by 
operational  restraints,  for  example  because  of  a  need  to  operate  the  system  covertly. 

Work  completed  by  Haduch  [7]  suggests  that  the  quality  of  feedback  provided  to  an  operator  will  effect  how 
the  UMV  is  operated.  Of  all  the  senses,  visual  feedback  is  most  often  favored  by  system  designers  for 
feedback,  possibly  because  it  is  perceived  as  being  the  most  informative.  Haduch  suggests  that  it  may  be 
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possible  to  utilise  a  form  of  cross-modal  communication;  using  audio  communication  rather  than  visual  to 
represent  speed  for  example.  Haduch  also  suggests  ways  of  reducing  the  communications  bandwidth  required 
to  operate  a  UMV.  A  simple  example  of  this  is  reproducing  vehicle  generated  sounds  at  the  control  station. 
This  way,  the  actual  data  of  the  sound  of  the  wheels  spinning,  for  example,  need  not  be  sent  (as  this  could 
place  heavy  demands  on  the  bandwidth  available),  but  merely  a  message  indicating  to  the  control  station  to 
make  a  sound  of  wheels  spinning.  Such  a  system  may  not  be  applicable  to  all  situations,  but  will  lessen  the 
need  for  large  communication  bandwidths  in  many  situations. 

Bandwidth  requirements  will  also  vary  depending  on  the  operators’  role.  A  supervisory  role  with  intervention 
from  the  operator  only  in  emergencies,  will  cope  much  better  with  smaller  bandwidth  communications  links 
than  a  system  that  requires  full  operator  in  the  loop  control.  This  implies  greater  implementation  of 
automation  into  UMVs  which  in  turn  has  human  factors  considerations  (e.g.,  see  above),  but  will  lessen 
bandwidth  requirements. 

Any  bandwidth  reduction  would  initially  require  investigation  into  what  the  key  information  needed  for  the 
task  was.  This  information  is  relatively  lacking  in  the  literature,  although  this  is  most  probably  because  the 
key  information  will  the  highly  task  dependant  [6]. 
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6.8  SUMMARY 

This  chapter  described  many  candidate  control/display  technologies  that  are  potentially  applicable  to  UMV 
operator  interface  design  and  detailed  other  important  design  considerations.  However,  since  UMVs 
encompass  a  very  broad  range  of  vehicle  types,  capabilities,  mission  contexts,  and  environmental  constraints/ 
affordances,  it  is  important  that  any  UMV  operator  interface  design  follow  a  multi-disciplinary  user-centered 
design  process.  The  goal  of  user-centered  design  is  to  ensure  the  final  design  meets  the  users’  needs  and 
expectations.  The  process  of  requirements  definition  (user  profiles,  work  flow,  task  analysis,  and  information 
architecture)  and  repeated  interface  design  development  and  iteration  (through  multiple  usability  assessments 
and  formal  evaluations)  will  increase  the  likelihood  of  obtaining  a  truly  functional  and  easy-to-use  interface. 

Other  key  conclusions  can  be  drawn  from  this  chapter,  besides  the  importance  of  user-centered  design. 
First,  the  potential  importance  of  mission-specific  multi-modal  interfaces  to  UMVs  is  highlighted.  Since  UMV 
operators  are  currently  limited  to  a  reduced  stream  of  sensory  feedback  delivered  almost  exclusively  through 
the  visual  channel,  there  is  reason  to  believe  that  situation  awareness  and  performance  may  be  improved 
through  increased  multi-sensory  stimulation.  These  improvements  might  stem  from  an  increase  in  the 
operator’s  sense  of  ‘presence’  in  the  remote  environment,  from  increased  information  throughput  afforded  by 
effective  use  of  multi-sensory  stimulation,  and/or  a  more  intuitive  presentation/control  of  information,  and 
thus  improved  performance  over  conventional  visual  interface  practices.  Technologies  such  as  spatialized 
audio,  haptic/tactile  stimulation,  and  speech  recognition  systems  appear  especially  relevant  to  multi-UMV 
operations. 

Additionally,  as  UMVs  become  more  automated/autonomous  and  single  operator  control  of  multiple  UMVs  is 
mandated,  the  importance  of  the  human-automation  interface  becomes  paramount.  As  mentioned  above  and  in 
Chapter  7,  there  is  a  wealth  of  potentially  negative  effects  associated  with  human-automation  systems  that 
have  significant  implications  for  the  operator  interface  design.  Automation  must  be  designed  to  augment,  not 
hinder,  human  capabilities.  Operator  interfaces  must  provide  rapid  visibility  into  the  current  status  and  future 
plans  of  automation  for  shared  human-automation  situation  awareness.  Additionally,  intelligent  decision 
support  interfaces  will  need  to  be  designed  such  as  to  allow  independent  operator  assessment  of  the  situation 
as  well  as  the  rationale  for  any  automated  classifications/recommendations. 

Finally,  the  chapter  clearly  indicates  several  areas  in  need  of  further  research.  These  areas  include  the  relative 
costs/benefits  of  the  various  control/display  technologies  identified  for  particular  classes  of  UMV  applications 
(air,  ground,  sea),  a  better  understanding  regarding  the  application  of  multi-modal  interface  technology  for 
UAV  operator  interfaces,  the  issues  surrounding  human  supervisory  control,  and  the  design  of  effective 
decision  support  interfaces  for  enabled  multi-UMV  control. 
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Chapter  7  -  HUMAN  AUTOMATION  INTEGRATION 

Chapter  Lead:  S.  Galster 

Contributors:  M.  Barnes,  K.  Cosenzo,  S.  Galster,  E.  Hollnagel, 

C.  Miller,  R.  Parasuraman,  J.  Reising,  R.  Taylor,  L.  van  Breda 

7.1  INTRODUCTION 

Many  versions  of  future  concept  of  operations  (CONOPS)  rely  heavily  on  UMVs.  The  pressure  to  take  the 
human  out  of  immediate  control  of  these  vehicles  is  being  driven  by  several  factors.  These  factors  include  a 
reduction  in  cost  for  the  production  and  maintenance  of  the  vehicle,  operational  viability  in  extreme 
environments,  and  the  public  pressure  to  keep  soldiers  further  away  from  immediate  harm.  In  addition  to 
adding  more  UMVs,  there  is  also  a  push  to  have  these  vehicles  perform  more  complex  tasks  than  they  are 
currently  required  to  perform.  These  two  factors,  adding  more  UMVs  and  having  them  perform  more  complex 
tasks,  will  not  be  realized  without  augmenting  the  current  structure  of  control.  One  way  to  achieve  this 
augmentation  is  through  the  utilization  of  automation.  The  automation  may  be  applied  on  the  vehicle  itself, 
through  the  interface  controlling  the  vehicle,  through  system  design  or  in  any  amalgamation  of  these 
approaches.  Automation,  if  applied  in  a  responsible  and  judicious  manner,  will  enable  the  acquisition  of 
capabilities  that  will  be  required  to  operate  under  near  and  far- term  CONOPS. 

The  focus  of  this  chapter  is  the  discussion  of  the  past,  present  and  future  automation  integration  challenges 
that  are  faced  when  adopting  a  human  centric  design  philosophy.  Topics  will  include  the  identification  and 
discussion  of  specific  problematic  areas  that  have  evolved  and  will  bring  to  bear  the  lessons  learned  as 
automation  was  integrated  into  other  domains  such  as  flight  operations,  air  traffic  control,  and  process  control 
for  example.  The  lessons  learned  may  not  signify  that  a  particular  problem  area  has  been  “solved”,  but  may 
point  out  that  the  area  deserves  consideration  when  evaluating  trade-offs  in  the  system  design  and  engineering 
process.  The  anecdotal,  operational,  theoretical  and  empirical  work  completed  thus  far  all  provide  a  sound 
foundation  that  should  serve  as  a  starting  point  for  human  automation  integration  in  the  UMV  domain. 
This  domain  may  have  specific  challenges  or  specific  opportunities  available,  both  now  and  in  the  future, 
to  explore  and  expand  the  base  of  human  and  automation  integration  knowledge.  The  remainder  of  this 
introduction  will  focus  on  some  of  the  more  salient  areas  in  the  integration  problem  space.  These  topics  and 
others  will  be  addressed  in  various  sections  throughout  the  chapter. 

7.1.1  Problems  with  Supervisory  Control  Tasks 

Vehicle  control  may  be  considered  as  a  hierarchically  structured  set  of  functions.  Plan  conception  and  plan 
selection  activities  are  performed  in  the  navigation  function,  verification  and  adjustment  of  the  short-term 
voyage  progress  are  performed  in  the  guidance  function,  and  typical  closed-loop  control  activities  are 
performed  in  the  control  function.  Supervisory  control  of  vehicles  deals  with  automated  vehicle  control 
functions  to  a  large  extent.  Current  technology  is  not  yet  prepared  to  autonomously  perform  mission  guidance 
under  dynamically  changing  environmental  conditions,  although  navigation,  guidance  and  control  functions 
are  widely  automated.  The  operator,  who  may  observe  the  controlled  process,  acts  as  a  manager  who 
supervises  the  system  and  interacts  with  the  automated  system  by  performing  corrective  actions.  The  human 
operator  in  current  architectures  is  the  primary  responsible  factor  in  terms  of  goal-driven  decision-making,  and 
thereby  specifying  the  constraints  and  demands  settings  for  the  automated  system. 
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It  is  known,  however,  that  supervisory  control  systems  have  certain  limitations  in  performance,  either  on  the 
operator’s  side  due  to  human  capacity  limitations  or  induced  by  deficiencies  of  the  automation,  causing 
human  error  intensified  by  the  inability  of  the  automation  to  perform  on  the  higher  level  of  problem-solving. 

7.1.2  Function  Allocation 

Consider  the  development  of  human  roles  and  automation  from  the  traditional 66 Left-over  principle ”,  through 
human  engineering  optimising  compensatory  principle  with  human  monitoring  (Fitts  lists),  to  contemporary 
complementary  principle  arising  from  human-computer  co-operation/collaboration.  Now  function  allocation 
can  be  dynamic  according  to  external  system  functions,  efficiency  and  system  boundary  conditions.  Levels  of 
autonomy  provide  bounding  of  the  decision  authority  and  behaviour  to  promote  trust.  Key  questions: 

•  Should  the  human  monitor  the  (technical)  system  given  that  humans  are  poor  monitors? 

•  Should  the  (technical)  system  monitor  the  human? 

•  If  so  what  roles  should  the  human  play  and  what  are  their  responsibilities? 

•  Are  humans  included  in  systems  just  to  deal  with  those  functions  that  engineers  can  not  automate? 

7.1.3  Levels  of  Automation 

For  designing  supervisory  control,  possible  structures  for  the  allocation  of  decision-making  tasks  between  human 
and  computers  are  complex  (up  to  10  levels).  These  have  been  applied  to  stage  models  of  human  information 
processing  functions  (information  acquisition,  analysis,  decision  selection,  action  implementation). 

7.1.4  Pilots  Associate  Levels  of  Autonomy 

Autonomy  can  be  defined  simply  as  the  capability  to  make  decisions.  Pilots  Associate  (PA)  Levels  of  Autonomy 
(LOA)  and  prime  directives  recognise  the  need  for  the  operator/pilot  to  remain  in  charge.  PA  provides  selectable 
LOA  (Inactive,  Standby,  Advisor,  Assistant,  Associate)  defined  by  operational  relationships  with  bounded 
structure,  but  this  needs  to  be  readily  communicable. 

7.1.5  Cognitive  Cockpit  PACT 

User  knowledge  acquisition  produces  a  practical,  communicable  set  of  assisted  PACT  levels  (At  Call, 
Advisory,  In  Support,  Direct  Support)  for  variable  and  adaptive  decision  support/automation,  supporting 
situation  assessment,  decision  making  and  action.  Research  focus  shifts  to  developing  practical  procedures  for 
pre-assigning  functions  and  tasks,  operator  initiated  real-time  changes,  and  triggering  level  changes  from 
context-sensitive  adaptive  rules.  Scripts  and  play -book  tools  for  delegation  of  tasks/policy  become  relevant. 

7.1.6  UAV/UCAV  Autonomy 

PACT  LOA  have  been  applied  to  the  management  of  multiple  UAVs  to  help  pilot/ground  operator  workload. 
DARPA  UAV  programmes  use  four  levels  of  autonomy  with  intermediate  levels  of  exception  and  consent 
(UCAV),  and  veto  and  permissive  (ICAV).  UCAV  research  focus  moves  to  multiple  collaborating, 
autonomous  groups  covering  complimentary,  co-ordinated  and  co-operative  planning  and  interactions. 


7-2 


RTO-TR-HFM-078 


dMNATO 
WP  I  OTAN 


HUMAN  AUTOMATION  INTEGRATION 


7.1.7  Multi- Agent  Adjustable  Autonomy 

Dynamic  adaptive  and  adjustable  autonomy  is  proposed  for  multi-agent  intelligent  systems  for  distributed 
problem  solving  structures  in  complex  dynamic  environments.  Agents  have  self-direction  and  goals  with 
capability  to  form,  modify  or  dissolve  the  agent  organisation.  Degree  of  autonomy  becomes  linked  to  individual 
goal.  Focus  moves  to  the  decision  process  for  how  a  goal  is  pursued  free  from  intervention,  oversight  or  control 
by  another  agent.  Autonomy  with  respect  to  goals  is  on  a  variable  scale  (consensus,  master,  local,  command). 
Issues  become  rules  for  transfer  of  control,  communication  protocols,  interaction  styles,  and  cognitive  strategies 
for  reasoning  with  adjustable  autonomy  in  operating  context. 


7.2  AUTOMATION  AND  HUMAN  PERFORMANCE 

Through  the  introduction  of  automation,  the  supervision  of  multiple  functions  is  more  and  more  assigned  to  a 
single  human  operator,  the  ‘process  manager’  or  ‘supervisor’,  who  is  assisted  by  a  process  information 
system.  Consequently,  the  level  of  direct  involvement  of  the  human  operator  with  the  actual  process  is 
decreasing.  It  must  be  realised  that  there  is  ample  evidence  that  lack  of  attention  to  the  human  aspects  in  early 
phases  of  the  development  process  of  a  system  may  result  in  wrong  usability.  It  is  therefore  essential  to 
consider  in  the  pre-design  phase,  to  what  extent  the  machines  should  be  made  automatic  and  how  this  affects 
human  factors  related  issues,  with  the  purpose  to  minimise  the  risk  of  errors  and  to  optimise  system 
performance.  The  following  deals  with  these  human  factors  considerations,  showing  the  areas  that  deserve 
attention. 

7.2.1  Humans  and  Unmanned  Military  Vehicle 

Current  UMVs  can  operate  in  a  more  or  less  autonomous  way.  This  means  that  the  operator-supervisor  is  no 
longer  directly  operating  the  UMV  directly,  but  parts  of  the  work  (semi-autonomous  or  semi-automatic)  or  the 
entire  job  (fully  automatic  or  autonomous)  are  operated  by  the  machine  itself.  In  the  consideration  of  semi¬ 
automatic  operation,  there  may  be  various  levels  of  human  involvement.  In  semi-automatic  systems,  the  system 
may  address  certain  sub-tasks  which  are  normally  operated  independently  (e.g.,  with  one  command  take-off  and 
go  to  cruising  altitude).  As  a  next  step,  the  machinery  could  perform  complete  well-defined  tasks  (e.g.,  perform  a 
battle  damage  assessment  task).  As  a  next  step,  a  more  complicated  set  of  tasks  may  be  executed,  such  as  fly 
back  to  base.  Finally,  a  complete  ‘mission’  may  be  run  automatically.  It  may  be  clear  that  fully  automatic 
systems  may  be  very  accurate  in  performing  repetitive  tasks.  Yet,  they  typically  lack  flexibility. 

At  present,  much  effort  is  spent  to  the  development  of  real-time  process  control  systems  as  a  means  to 
increase  the  efficiency  and  the  safety  of  automated  systems.  It  is,  however,  important  to  maintain  an  operator- 
centred  automation  philosophy  that  overcomes  limitations,  enhances  abilities  and  fosters  acceptance  when 
automated  systems  are  introduced.  Therefore,  focus  on  human  factors  issues  related  to  the  automation  is 
essential.  Below,  a  generic  list  of  issues  is  provided  together  with  criteria  that  should  be  taken  into  account 
when  automation  is  introduced. 

In  general,  the  user-centred  design  process  starts  with  a  function  analysis,  function  allocation  and  task  analysis 
process,  in  which  the  demands  that  UMV  moving  tasks  put  on  the  user,  the  identification  of  information  to  be 
processed  and  decisions  to  be  made  by  the  user,  are  determined.  The  opportunities  provided  by  new 
technologies  in  performing  UMV  tasks  have  to  be  analyzed  in  terms  of  where  and  how  they  could  support  the 
human  operator.  For  this  purpose,  a  generic  model  of  information  transfer  must  be  developed,  which  can  be 
used  to  support  the  analysis.  On  the  basis  of  this  generic  model,  relevant  issues  are  described  which  focus  on 
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human  factors  aspects  with  respect  to  control  and  supervision  of  (semi-)  autonomous  systems.  This  must  lead 
to  a  list  of  recommendations.  Throughout  the  report,  the  relation  with  a  practical  situation  is  explained 
(i.e.,  UMV  operation). 

Technological  developments  more  and  more  allow  sub-systems  of  machinery  to  be  automated.  The  potential 
benefits  of  automation  express  themselves  in  different  areas:  it  can  make  the  machines  to  perform  the  work 
faster  and  more  accurate,  it  can  reduce  the  fuel  consumption  and  increase  the  economical  utilization  of  the 
machines.  Furthermore,  automation  can  relieve  the  human  operator  from  dull,  repetitive  or  dangerous  work, 
and  reduce  the  workload.  However,  there  are  also  potential  drawbacks  from  a  human  factors  point  of  view 
when  automation  is  introduced.  For  example,  in  the  cockpit  of  commercial  airlines  it  appeared  that  automation 
lowered  job  satisfaction,  induced  human  error,  and  caused  a  significant  loss  of  skilled  behavior.  Parasuraman 
and  Riley  [1]  give  a  summary  that  includes  the  use,  misuse,  disuse  and  abuse  of  automation.  Stokes,  Wickens 
and  Kite  [2,  p.  101]  stated  that ‘a  potential  danger  is  to  automate  tasks  that  are  easy  or  beneficial  to  automate, 
instead  of  those  tasks  which  are  eligible  for  automation  from  the  operator’.  This  means  that  sometimes  the 
overall  performance  of  a  system  may  be  enlarged  by  allocating  a  certain  function  to  a  human,  although  an 
automaton  would  perform  better  on  the  specific  function.  This  implies  that  evaluating  the  system  as  a  whole 
with  the  operator  in-the-loop  is  at  least  as  important  as  evaluating  the  performance  of  individual  automated 
subsystems,  “designing  or  automating  a  human-machine  system  rarely  produces  an  acceptable  result  without 
extensive  searches  through  alternative  designs  plus  experimentation  to  evaluate  overall  system  performance; 
there  are  no  shortcuts  to  success ”  [3,  p.  1459]. 

In  the  literature,  different  definitions  of  automation  are  used.  The  most  common  definition  is  used  by  Wiener 
[4],  Satchell  [5],  and  many  others:  ‘Automation  is  concerned  with  replacing  human  functioning  by  machine 
functioning’.  This  applies  for  both  partial  and  total  replacement  [6].  Despite  the  agreement  on  the  definition  of 
automation,  several  authors  distinguish  many  sorts  and  levels  of  automation.  Although  these  sorts  and  levels 
are  not  necessary  to  discuss  the  human  factors  involved  in  automation,  they  may  be  useful  in  understanding 
the  conceptual  frameworks  and  the  complexity  of  the  underlying  knowledge  domain.  Wickens  [7] 
distinguished  automation  for  replacing  the  human,  and  automation  for  supporting  the  human;  Wiener  [4] 
distinguished  automation  of  control  tasks,  and  automation  of  monitoring  functions  (which  can  be  divided  into 
automation  of  detection,  and  automation  of  diagnosis).  These  distinctions  are  based  on  the  different  levels  of 
human  functioning,  and  resemble  the  approach  of  Sheridan  [8].  The  latter  bases  his  conceptual  framework 
upon  the  three  levels  as  defined  by  the  model  of  Rasmussen  [9]:  Skill  based,  rule  based,  knowledge  based. 
Sheridan  claims  that  for  successful  automation,  one  has  to  start  at  the  skill  based  level;  automation  of  tasks  at 
the  knowledge  based  level  is  exceptional. 

Although  the  introduction  of  automation  sometimes  follows  on  the  availability  of  new  techniques  and 
technological  developments,  it  is  also  recognized  that  there  are  important  potential  advantages  for  the  operator 
and  the  system  output  when  human  functions  are  transferred  to  a  machine.  Automatons  can  simply  be  better  in 
certain  tasks,  for  example,  to  perform  route  planning  tasks  in  order  to  minimize  fuel  usage  [10,4].  Even  simple 
automatons  may  be  faster,  more  accurate,  more  reliable,  less  stressful,  less  vulnerable  for  small  errors, 
and  cheaper  than  human  operators.  They  even  may  be  able  to  perform  tasks,  which  the  operator  is  able  to 
specify,  but  not  able  to  execute  (including  tele-operation).  Important  considerations  may  also  be  aimed  at 
relieving  the  human  from  time  consuming,  dangerous,  complex,  tiring  or  dull  (repetitive)  tasks  [2].  However, 
the  main  goal  concerning  the  human  operator  is  to  reduce  workload  [3,11].  This  may  enhance  both  the  well¬ 
being  of  the  operator,  and  the  performance  of  the  total  system.  There  are  additional  effects  of  automation, 
which  are  not  directly  the  goal  of  the  design  process,  but  offer  opportunities  to  further  improve  system 
performance  (e.g.,  automation  may  offer  possibilities  to  introduce  on-line  simulation  of  the  work  process; 
predictive  displays  may  be  used  to  support  the  anticipation  process;  ‘fail  soft’  protection  may  prevent 
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operators  and  equipment  to  commit  certain  errors,  limiting  the  influence  of  individual  differences  on  system 
performance). 

It  may  be  clear  that  the  role  of  the  human  operator  is  considered  trivial  by  some,  and  critical  by  others; 
the  fundamental  nature  and  the  characteristics  of  the  interaction  between  operator  and  automated  system  is 
still  a  matter  of  debate  and  design  evolution.  This  introduction  underlines  the  importance  of  considering  the 
human  factor  when  automatic  systems  are  introduced. 

7.2.2  Human  Factors  and  Automation 

Automation  may  be  introduced  to  replace  or  support  operator  functions  at  different  levels.  In  the  literature, 
various  sub-divisions  are  made,  ranging  from  ‘no  automation’  to  ‘fully  autonomous  operation’.  A  useful 
subdivision  was  made  in  AGARD  [6]  with  seven  different  levels  of  automation  (i.e.,  no  automation,  manual 
augmented,  manual  augmented  limited,  co-operative,  automatic  pre-select,  automatic  select,  and  autonomous 
operation).  In  this  Introduction,  these  levels  of  automation  are  applied  to  UMVs.  An  eighth  level  (unmanned 
operation)  is  added.  Note  that  unmanned  operation  is  possible  at  all  levels  of  automation,  however,  this  is  not 
considered  within  the  scope  of  this  Introduction.  Practical  applications  are  suggested  for  each  level  of 
automation,  and  the  implications  with  respect  to  human  factors  considerations  are  discussed. 

1)  No  Automation 

The  system  is  manually  operated  without  any  automatic  augmentation  or  support.  The  operator  is 
performing  activities  using  his  human  faculties  (e.g.,  visual  functions,  mental  activities,  visual  inspection, 
verbal  communication).  The  control  movements  by  the  operator  directly  affect  system  output. 

2)  Manual  Augmented 

Manual  control  is  augmented  by  an  automatic  system  when  an  automatic  system  assists  the  operator  by 
controlling  simple  activities  (so-called  low  level  activities).  Examples  are  augmented  systems  that  control 
a  single  system  variable  (e.g.,  speed,  frequency,  rpm,  strength,  pressure),  or  provide  low  level  decision 
support  (e.g.,  records  images  when  desired). 

3)  Manual  Augmented  and  Limited 

Manual  control  is  augmented  and  limited  when  an  automatic  system  assists  the  operator  by  controlling 
low  level  activities  such  that  over-control  and  control-errors  are  avoided.  An  example  is  the  anti-lock 
brake  system  (ABS)  in  cars;  the  driver  needs  less  pedal  force  to  execute  a  brake  action,  and  moreover, 
when  a  full  stop  manoeuvre  is  made,  the  braking  action  is  limited  in  order  to  avoid  break-out  of  the  car. 
Another  example  is  the  speed-limiter  in  trucks;  the  truck  will  stop  acceleration  when  a  certain  speed  is 
reached,  even  when  the  gas  pedal  is  pushed  down  further.  An  example  in  decision  making  support  is  the 
monitoring  of  input  sequences  (e.g.,  a  car  does  not  start  the  engine  as  long  as  it  is  put  into  gear). 

4)  Co-operative 

Co-operative  support  of  manual  control  is  an  automatic  function  with  pre-selection  of  parameters  and  their 
values,  in  combination  with  manual  control.  An  example  is  dynamic  positioning  of  a  ship  during 
disturbances  (e.g.,  in  wind  and  water  current).  Another  example  is  cruise  control  in  cars;  the  driver  selects 
the  desired  speed  and  the  car  will  drive  at  that  speed,  irrespective  of  the  road  condition  (e.g.,  highway  or 
desert  road,  hill  upward  or  downward).  Note  that  co-operative  support  may  be  superimposed  upon  the 


RTO-TR-HFM-078 


7-5 


HUMAN  AUTOMATION  INTEGRATION 


ORGANIZATION 


lower  support  levels  to  enable  the  operator  to  go  beyond  the  installed  limits  when  certain  extreme  control 
conditions  are  met. 


5)  Automatic  Pre-Select 

Support  of  manual  control  by  automatic  pre-select  is  an  automatic  function  with  pre-selection  of 
parameters  and  their  values,  without  manual  control.  In  fact,  automatic  pre-select  may  be  conceived  as  the 
replay  of  a  set  of  pre-programmed  actions.  An  example  is  automatic  course  control  (auto-pilot  control) 
on  board  ships;  the  navigator  selects  a  new  course,  starts  the  course  change  manoeuvre  by  pressing  a 
push-button,  and  the  vessel  will  automatically  alter  course  with  a  pre-selected  turn  rate.  Note  that  pre¬ 
select  automation  can  be  superimposed  upon  the  lower  support  levels  (e.g.,  the  navigator  cannot  execute 
manoeuvres  with  course  changes  of  over  180  degrees). 

6)  Automatic  Select 

Support  of  manual  control  by  automatic  select  is  an  automatic  function  that  performs  certain  functions 
automatically.  These  functions  can  be  selected  or  deselected  (switched  on  or  off)  by  the  operator. 
An  example  is  an  automatic  track  keeper  on  a  ship:  The  navigator  selects  navigation  points  (way  points) 
to  follow,  enables  automatic  select  mode  by  pressing  a  push-button,  and  the  vessel  will  determine  the 
optimal  track  between  the  two  navigation  points  (considering  the  effect  of  water  current,  wind,  and  local 
weather  by  means  of  databases),  and  then  start  to  follow  this  optimal  track.  The  navigator  only  has  to 
accept  (correct,  or  reject)  the  proposed  track. 


7)  Autonomous  Manned  Operation 

During  autonomous  manned  operation,  the  work  is  done  automatically.  The  operator  is  still  present  on  the 
vehicle,  monitoring  the  system  and  surveilling  the  execution  of  the  work  procedure. 


8)  Autonomous  Unmanned  Operation 

The  work  is  done  autonomously.  There  is  no  operator  on  the  vehicle  itself. 

The  effect  of  these  levels  of  automation  on  the  human  factors  of  UMVs  is  roughly  evaluated  below. 
For  this  purpose,  the  effects  of  different  levels  of  automation  on  the  main  human  factors  related  issues 
were  assessed  by  the  author.  Table  7-1  shows  a  summary  of  the  results.  In  the  Table,  a  +  was  rated  when  a 
positive  effect  on  the  related  issue  was  expected,  a  ++  when  the  effect  was  rated  very  positively.  A  -  was 
rated  when  a  negative  effect  on  the  related  issue  was  expected,  a  -  when  the  effect  was  rated  very 
negatively.  The  assessment  is  made  subjectively  by  the  author,  based  on  the  author’  expertise  in  human 
factors  and  on  the  information  derived  from  the  literature.  It  has  to  be  stressed  that  some  items  in  the 
assessment  are  disputable,  but  the  overall  picture  gives  an  indication  of  the  advantages  and  drawbacks  of 
the  different  automation  levels.  All  the  arguments  that  led  to  Table  7-1  are  explained  hereafter. 
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Table  7-1:  Evaluation  of  the  Application  of  Automation  on  Unmanned  Military  Vehicles  (UMVs) 


Automation  Level 

Human  Factors  Related  Issues 

System  Output 

Peripheralization: 
Cognitive  Issues 

Peripheralization: 

Control  Issues 

General  Issues 

Task 

Involvement 

i.i ') 

Vigilance 

1.2 

Direct 

Control 

2.1 

Maintaining 

Skills 

2.2 

Maintaining 

Awareness 

2.3 

Few 

Mental 

Resources 

Required 

3.1 

Relief  of 
Physical 
Workload 
3.2 

Comfort 

3.3 

Little 

Training 

Required 

3.4 

Output 

Quality 

Economical 

Usage 

1)  No  Automation 

+  + 

+  + 

+  + 

+  + 

+ 

+  + 

- 

+  + 

2)  Manual  Augmented 

+  + 

+  + 

+  + 

+  + 

+ 

+  + 

- 

- 

- 

+  + 

3)  Manual  Augmented 
Limited 

+  + 

+  + 

+ 

++ 

+ 

+  + 

- 

- 

- 

+  + 

- 

4)  Co-operative 

+ 

+  + 

- 

+ 

- 

+  + 

+ 

+ 

+ 

+ 

+ 

5)  Automatic 

Pre-Select 

+ 

+  + 

- 

+ 

- 

+  + 

+ 

+ 

+  + 

+ 

+  + 

6)  Automatic  Select 

- 

+  + 

- 

- 

- 

+  + 

+ 

+ 

+  + 

+ 

+  + 

7)  Autonomous 

Manned 

- 

+ 

+  + 

+  + 

- 

+  + 

- 

+  + 

8)  Autonomous 
Unmanned 

- 

- 

+  + 

+  + 

+  + 

+ 

- 

+  + 

Note: 


+  = 


very  negative  effect 
negative  effect 
positive  effect 
very  positive  effect 
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As  indicated  in  Table  7-1,  task  involvement  (1.1)  -  a  low  state  increases  the  risk  of  complacency  (i.e.,  a  low 
index  of  suspicion  due  to  automation)  -  is  significant  at  lower  levels  of  automation.  The  operator  is  (nearly) 
solely  operating  the  UMV,  directly  controlling  airframe  and  sensors.  The  operator  is  closely  ‘in-the-loop’, 
that  is,  he  is  continuously  involved  in  manual  control  activities,  and,  as  well,  continuously  physically 
influenced  by  the  behavior  of  the  vehicle.  The  more  automatic  equipment  is  installed,  the  more  the  operator 
will  rely  on  this,  which  increases  the  risk  of  complacency.  In  automation  levels  6-9  there  are  situations  that 
complete  functions  will  be  taken  over  by  the  machine,  resulting  in  a  low  degree  of  task  involvement  (i.e.,  high 
risk  of  complacency). 

Vigilance  (1.2),  defined  as  the  ability  to  detect  infrequent  signals  over  prolonged  periods  of  time,  is  high  when 
the  operator  actively  participates  in  the  process.  At  automation  levels  1-6  there  are  still  a  number  of  manual 
activities  to  perform  by  the  operator.  This  is  hardly  the  case  during  autonomous  operation  (levels  7  -  8), 
when  nearly  all  operator  functions  are  performed  by  the  automated  systems;  however,  the  negative  effect  on 
vigilance  will  be  limited  in  these  situations  when  adequate  feedback  of  the  process  state  is  provided  to  the 
operator. 

In  the  lower  levels  of  automation  (levels  4-6)  there  is  a  direct  link  between  operator  control  actions  and 
process  control  activities.  In  case  of  high  level  automation  (levels  7-8)  there  is  a  direct  link  between 
automatic  system  actions  and  process  control  activities.  In  both  cases  it  is  well  possible  to  match  input  and 
output,  which  has  a  positive  influence  on  direct  control  (2.1).  In  semi-automated  operation  (levels  4  -  6), 
control  activities  are  performed  by  operator  and  by  machines  simultaneously.  It  is  then  well  possible  that  the 
sequencing  of  the  output  activities  is  conflicting;  automatic  functions  will  be  executed  more  or  less 
independently  from  actions  taken  by  the  operator  (e.g.,  it  is  known  that  pre-select  parameters  may  be  changed 
during  execution  just  to  fool  the  automatic  system  in  order  to  speed  up  the  sequencing,  or  to  start  the  sequence 
earlier).  This  has  a  negative  influence  on  direct  control. 

Automation  has  a  drawback  on  maintaining  skills  (2.2),  that  is,  the  ability  to  perform  the  manual  task  properly 
when  situations  require  switching  back  from  automatic  to  manual  control.  Literature  indicates  that,  the  more 
automatic  systems  there  are  introduced,  the  more  the  operator  will  be  placed  out-of-the-loop.  A  lack  of 
operator  involvement  is  the  result,  with  increasing  risk  in  loss  of  skill.  Particularly  in  the  design  of 
autonomous  systems  this  deserves  attention. 

About  operator  awareness  (2.3),  defined  as  the  ability  to  perceive  elements  in  the  environment  and  to 
understand  their  meaning,  it  is  stated  that  the  operator  is  fully  aware  of  machine  state  and  controlled  process 
state  in  case  there  is  a  high  degree  of  direct  control  (see  2.1,  the  levels  1-3,  and  the  levels  7  -  8),  provided 
that  there  is  adequate  feedback  of  machine  and  process  state.  However,  in  levels  1  -  3  it  may  be  expected  that 
the  operator  is  not  fully  aware  of  what  the  impact  of  his  actions  be  on  a  number  of  activities  (e.g.,  does  his 
activity  still  fit  in  the  work  plan;  what  are  the  actual  costs;  are  there  time  delays;  are  there  changes  in  the  work 
plan).  Worst  operator  awareness  is  rated  when  a  part  of  his  activities  is  taken  over  by  automated  systems 
(i.e.,  items  4  -  6).  Machine  and  operator  are  performing  their  assigned  activities,  with  the  risk  that  the  machine 
performs  activities  that  are  not  (or  insufficiently)  perceived  by  the  operator/supervisor.  For  example,  in  case  of 
an  automatic  visual  target  tracking  system,  it  is  important  to  have  the  system  dead-reckoning  when  the  target 
is  obscured.  Again,  adequate  feedback  is  essential  in  this  case. 

Part-task  automation  (levels  1-6)  has  a  positive  effect  on  the  required  mental  resources  (3.1)  (i.e.,  routine 
tasks  require  few  mental  resources,  resulting  in  low  mental  workload).  The  operator  still  directly  perceives 
(e.g.,  by  force  feedback,  auditory/vibration  feedback)  the  system’s  activities,  and  few  mental  transformations 
have  to  be  made  to  understand  what  is  going  on,  and  what  (manual)  activities  have  to  be  performed.  It  is 
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assumed  that  more  operator  attention  is  required  during  autonomous  operation,  particularly  in  an  unmanned 
situation.  The  operator  is  then  tele-operating  the  system,  while  collecting  information  only  through  feedback 
systems.  In  that  case  it  is  expected  that  more  mental  transformations  are  needed  (e.g.,  engine  rpm  has  to  be 
verified  by  means  of  an  indicator  on  a  screen,  instead  of  interpreting  engine  noise/vibration). 

Physical  workload  (3.2)  -  the  biomechanical  conditions  of  the  operator  during  task  execution  is  fully  relieved 
during  autonomous  unmanned  operation  when  the  operator  is  located  in  a  remote  station.  There  will  be  some 
relief  in  case  certain  functions  are  taken  over  by  automatons  (levels  4  -  7);  no  relief  of  physical  workload  is 
expected  when  no  automation  is  installed  (level  1),  or  when  the  operator  is  manning  an  autonomous  system 
(level  7).  In  the  latter  case,  the  operator  is  sitting  on  board  a  machine  that  performs  activities  autonomously, 
most  of  which  the  operator  is  not  aware  of  or  even  may  not  expect. 

Comfort  (3.3),  in  the  sense  of  physical  workload,  is  rated  low  for  low  level  of  automation  (levels  1  -  3), 
and  medium  for  part-task  automation  (levels  4  -  6).  Comfort,  in  the  sense  of  being  informed  about  the  actual 
status  of  the  total  system,  is  rated  very  low  in  case  of  manned  autonomous  operation  (level  7).  The  machine 
will  perform  actions  that  are  not  attended  by  the  operator,  who  is  sitting  in  the  cabin. 

Little  training  (3.4)  is  required  in  situations  of  part-task  and  full  automation  (i.e.,  automation  levels  4  -  7). 
Note  that  training  in  autonomous  unmanned  operation  (level  8)  may  include  tasks  that  are  not  relevant  in  the 
context  of  the  other  automation  levels  (1-7). 

System  output  quality  is  rated  positive  when  few  operator  functions  are  automated.  The  more  automatic 
systems  are  used,  the  less  flexible  the  system  will  be.  In  case  of  part-task  automation  conflicts  between 
operator  actions  and  automatic  system  activities  are  likely  to  happen;  in  case  of  autonomous  operation  the 
sequencing,  the  function  execution  mostly  is  pre-programmed.  In  contrast  to  this,  economical  usage  of  the 
system  will  increase  the  more  automatic  systems  are  used  (i.e.,  use  on  a  continuous  basis;  fine-tuned  in  energy 
consumption  for  each  job;  better  maintenance  scheduling,  less  influence  of  environmental  disturbances,  etc.). 

7.2.3  Recommendations 

The  following  recommendations  with  respect  to  automation  are  given: 

1)  Select  the  proper  level  of  automation 

Each  level  of  automation  has  its  specific  implications  on  the  human  factors  aspects.  When  considering 
Table  7-1,  a  number  of  conclusions  can  be  drawn.  Low  level  of  automation  (levels  1-3)  results  in  best 
quality  of  work  and  will  not  lead  to  operator  peripheralization.  However,  the  operator  then  performs  under  a 
high  degree  of  physical  load  which  limits  the  economical  usage  of  the  equipment.  Medium  level  of 
automation  (levels  4  -  6)  is  rated  most  positively.  It  is  essential,  however,  that  adequate  feedback  of  the 
system  state  is  provided  to  the  operator  so  that  optimal  awareness  is  maintained  (e.g.,  by  providing  visual 
display  information  and  haptic  feedback  in  the  controls).  The  operator  is  kept  in-the-loop  which  benefits  the 
quality  of  task  execution.  Total  (autonomous)  operation  is  only  beneficial  in  an  unmanned  configuration. 

2)  Keep  the  operator  in-the-loop,  increase  operator  awareness,  and  provide  adequate 
feedback 

The  operator  should  actively  participate  in  the  job.  Adequate  visual  feedback  on  displays  should  be  presented 
for  monitoring  purposes,  and  haptic  feedback  by  providing  active  controls  for  sensory  information.  Have  the 
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operator  activate  sequences,  or  confirm  the  actual  system  status,  on  a  regular  basis.  Avoid  simultaneous 
monitoring  and  manual  control  activities.  Avoid  discrepancies  between  indicators  and  actual  mode,  e.g.,  use 
pushbuttons  with  visual  indicators.  Introduce  dedicated  displays,  presenting  the  present  status,  the  oncoming 
actions,  and  the  current  plan.  The  above  mentioned  importance  of  feedback  is  confirmed  in  the  literature, 
including  lessons  learned  in  cockpit  automation,  and  in  generic  model  descriptions  of  a  human-machine 
interface  confirming  the  benefits  of  operator-in-the-loop  performance  [12].  Automation  may  result  in 
eliminating  or  replacing  critical  cues.  For  example,  specific  information  in  manual  control  tasks  (e.g.,  haptic 
information  in  joystick  controls)  is  needed  to  detect  system  state  changes  [13].  Norman  [14],  as  well  as 
Korteling  and  Van  Gent  [15],  stated  that  without  appropriate  feedback,  operators  are  indeed  out-of-the-loop: 
They  may  not  be  aware  whether  their  input  has  been  received,  whether  actions  are  performed  properly, 
and  what  the  intentions  of  the  system  are.  In  tracking  tasks,  error  detection  is  better  with  manual  control, 
due  to  the  presence  of  proprioceptive  feedback  [16].  Kessel  and  Wickens  [17]  hypothesized  that  the  fact  that 
subjects  operating  under  manual  control  had  both  visual  and  proprioceptive  feedback,  contributed  to  better 
performance.  The  need  for  appropriate  feedback  will  increase  with  increasing  complexity  and  with  increasing 
autonomy  of  the  system. 

3)  Introduce  flexible  automation 

The  operator  should  be  able  to  select  the  automated  activities  at  all  time.  This  puts  high  demand  on  the 
designers  of  the  automation  algorithms.  A  good  way  to  overcome  this  problem  is  to  have  the  operator  Team’ 
the  system  how  to  perform  the  individual  activities  (e.g.,  the  operator  first  shows  the  machine  how  to  equalize 
a  pile  of  material,  or,  how  to  put  a  load  at  certain  positions  when  performing  fork  work  activities). 

4)  Natural  operation 

The  system  should  perform  automatic  activities  as  if  this  was  done  by  the  operator  himself  during  automatic 
task  execution.  There  will  be  less  unattended  actions  of  the  system,  which  improves  the  operator’s  awareness 
and  comfort,  increasing  total  system  safety  and  performance.  Natural  operation  is  particularly  important  when 
the  operator  has  to  override  the  automatic  system  by  switching  back  to  manual  control. 

5)  Forestall  loss  of  skill 

The  best  way  to  prevent  the  loss  of  operator  skill  is  probably  to  periodically  give  the  operator  dedicated 
training.  Another  possibility  is  to  require  the  operator  to  perform  skill  critical  tasks  manually  at  certain  times, 
even  though  the  task  may  have  been  allocated  to  the  automated  system.  Furthermore,  the  use  of  active 
controls,  and  the  use  of  a  system  in  which  the  operator  Teams’  the  machine  how  to  perform  a  task  will  also 
help  to  prevent  skill  loss.  Both  these  options  were  mentioned  earlier. 

6)  Smooth  mode  transitions 

Enable  a  smooth  transition  between  manual  and  automated  control  mode,  e.g.,  prevent  sudden  forces  on 
controls.  In  fact,  this  implies  that  active  controls  must  be  installed.  The  operator  continuously  feels  what  the 
automatic  system  is  doing  just  by  putting  his  hands  on  the  controls.  Taking  over  manual  control  could  then  be 
just  a  matter  of  further  activating  that  control  (e.g.,  putting  pressure  on  the  grip,  or  activating  a  switch). 

This  Introduction  outlined  various  human  factors  considerations  in  the  use  of  automatic  systems.  It  indicated 
possible  advantages  and  disadvantages  of  the  various  levels  of  automation. 
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7.3  HUMAN  AUTOMATION  INTEGRATION  WITH  CONTRACTUAL 
AUTONOMY 

7.3.1  Introduction 

7.3. 1.1  Function  Allocation 

Automation  is  continually  improving  in  capability  with  associated  changes  in  perceptions  of  appropriate 
human  roles  and  the  suitability  of  functions  for  human  and/or  machine  performance.  Traditional  engineering 
mostly  used  the  “left  over”  principle  for  allocation  of  function,  where  the  technical  system  was  designed  to  do 
as  much  as  is  feasible  from  an  efficiency  point  of  view,  and  the  rest  was  left  for  the  operator.  HF  engineering 
introduced  the  compensatory  principle,  where  human  and  machine  capabilities  are  compared  on  salient 
criteria  and  the  function  allocation  is  made  so  that  the  respective  capabilities  are  used  optimally.  In  1951, 
Paul  Fitts  suggested  some  simple  criteria  for  allocating  functions  between  people  and  machines  to  predict 
roles  in  future  air  navigation  and  air  traffic  control  systems  [1].  Fitts  distinguished  between  four  kinds  of 
control  systems,  namely: 

1)  Fully  automatic  control; 

2)  Automatic  control  with  human  monitoring; 

3)  Semi-automatic  control  supplemented  by  human  performance  of  critical  functions;  and 

4)  Primary  control  by  human  operators. 

The  1994  NATO  RSG  workshop  on  function  allocation  [2]  reiterated  the  questions  posed  by  Fitts  and  his 
colleagues  which  still  lacked  general  answers: 

•  Should  the  human  monitor  the  (technical)  system  given  that  humans  are  poor  monitors? 

•  Should  the  (technical)  system  monitor  the  human? 

•  If  so  what  roles  should  the  human  play  and  what  are  their  responsibilities? 

•  Are  humans  included  in  systems  just  to  deal  with  those  functions  that  engineers  can  not  automate? 

Options  on  decision  making  were  noted  to  range  from  the  principle  that  the  human  should  make  all  decisions, 
because  humans  are  responsible  for  systems,  to  the  principle  that  there  are  some  decisions  that  humans  should 
never  be  permitted  to  make. 

In  the  1980’s,  with  increasingly  capable  intelligent  computing,  ideas  of  human-computer  teamwork,  cognitive 
engineering,  cognitive  automation  and  joint  cognitive  systems  began  to  emerge  [3].  According  to  the 
complementarity  principle,  function  allocation  serves  to  support  and  sustain  human  ability  to  perform 
efficiently.  Here,  the  focus  shifts  from  human-machine  interaction  to  human-computer  co-operation,  and  from 
the  internal  functions  and  structure  of  the  human  and  machine  to  the  external  functions  and  establishing  the 
system  boundaries  [4]. 

The  capability  for  collaborative  working  with  other  agents,  including  humans,  is  a  goal  for  intelligent 
automation.  Taylor  and  Reising  [5]  noted  that  in  order  to  work  collaboratively  with  humans,  intelligent 
automation  probably  requires  a  functional  architecture  with  the  following  attributes: 

•  A  model  of  human  decision  making  and  control  abilities. 
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•  The  ability  to  monitor  human  performance  and  workload  through  behavioural  and  physiological 
indices. 

•  The  ability  to  predict  human  expectations  and  intentions  with  reference  to  embedded  knowledge  of 
mission  plans  and  goals. 

7.3. 1.2  Automation  Reliability,  Trust  and  Use 

Issues  of  “trust”  have  featured  strongly  in  implementation  of  ideas  of  dynamic  function  allocation,  automated 
decision  support  and  human-computer  collaborative  adaptive  systems.  This  is  in  response  to  concerns  about 
human  monitoring  of  unreliable  automation  and  the  need  for  control  strategies  to  mitigate  bias  from 
complacency  and  over- trust,  or  alternatively,  under-trust  and  disuse  [6]. 

Early  UK  MOD  research  with  aircrew  investigated  the  structure  and  measurement  of  trust  to  help  design 
automation  safeguards.  A  study  of  twin-crew  RAF  Tornado  aircraft  operations,  elicited  tactical  decision 
making  scenarios  and  aircrew  rated  them  for  the  importance  of  factors  associated  with  trust  [7].  Demand  for 
trust  was  associated  with  perceived  risk  and  the  probability  of  negative  consequences,  whereas  the  supply  of 
trust  was  related  to  the  requirement  for  judgement  and  awareness,  and  uncertainty  and  doubt  in  making 
decisions.  Relying  on  others  to  make  risky  decisions  calls  for  a  large  amount  of  trust.  If  the  decision  requires 
another  person  exercising  a  high  degree  of  awareness  and  judgement,  and  there  is  much  uncertainty  and  doubt 
in  the  decision  provided,  then  the  actual  trust  engendered  by  the  decision  will  be  low.  In  a  follow-up  study  on 
the  quality  of  aircrew  teamwork  [8],  trust  was  found  to  be  a  significant  factor  in  distinguishing  between  good 
and  poor  teamwork  performance.  Trust  was  rated  at  a  significantly  lower  level  in  single-seat  RAF  Harrier 
operations  (i.e.,  human-computer  teamwork)  than  in  two-seat  RAF  Tornado  aircraft  tactical  operations 
(i.e.,  both  human-computer  and  human-human  teamwork). 

Several  models  of  trust  have  been  proposed.  Riley  [9]  developed  a  model  of  the  relationships  between  trust, 
operator  skill  level,  task  complexity,  workload  and  risk,  self-confidence  and  automation  reliability.  Studies  in 
which  workload  and  reliability  were  varied,  led  to  refinement  of  the  model  to  include  factors  of  fatigue  and 
learning  about  system  states  [10].  Further  research  has  modelled  trust  as  a  function  of  recent  performance, 
and  the  presence  and  magnitude  of  fault,  with  subjective  trust  increasing  with  automation  reliability  [11]. 
The  relationship  between  reliability  and  trust  was  confirmed  by  human  monitoring  performance  measures 
indirectly  measuring  trust  [12].  Recent  work  has  extended  modelling  to  investigate  how  the  dynamics  of  trust 
and  reliance  depend  on  information  sharing  [13],  and  how  trust  becomes  over-trust  through  unintended  use 
[14]. 

Experimental  evidence  has  verified  that  unexpected  automation  failure  leads  to  a  breakdown  of  trust,  and  to 
difficulty  in  the  recovery  of  trust  with  a  loss  of  faith  in  future  teamwork  performance.  As  trust  declines, 
manual  intervention  increases  [15,16].  Research  has  showed  how  when  workload  is  increased,  over-trust  or 
complacency  develops  with  automatic  systems,  and  coupled  with  vigilance  problems,  this  is  likely  to  lead  to 
failure  to  detect  performance  deviations  and  decrements  in  automation  performance  [17].  Operator  detection 
of  automation  failure  is  substantially  degraded  with  a  static  allocation  fixed  over  a  period  of  time,  favouring 
dynamic  adaptable  allocation  [18].  Manual  task  reallocation  has  been  proposed  as  a  countermeasure  to 
monitoring  inefficiency  and  complacency  since  short  periods  of  intermittent  manual  task  reallocation, 
or  cycling  between  manual  and  automation  control,  reduces  failures  of  monitoring  [19].  By  maintaining 
manual  skill  levels,  and  enhancing  situational  awareness,  manual  task  re-allocation  helps  when  intervention  is 
needed  following  automation  failure.  However,  without  active  involvement,  it  is  difficult  to  maintain  an 
appropriate  dynamic  internal  model  of  the  important  changing  relationships  needed  for  regaining  manual 
control  following  automation  [20].  Experimental  studies  have  shown  that  with  competing  demands  for 
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attention,  humans  are  poor  at  monitoring  automation  for  occasional  malfunctions,  exhibiting  automation 
complacency  [17].  Humans  also  have  poor  awareness  of  adaptable  automation  failure  [21,22]. 

Generally,  trust  is  best  considered  as  an  intervening  variable  between  automation  reliability  and  automation  use 
[6].  For  purposes  of  measuring  aiding  effectiveness,  like  “confidence”,  trust  operates  as  a  psychosocial  attitude 
with  inherently  variable  subjective  complexity,  rather  than  as  a  cognitive  functional  state  such  as  “situation 
awareness”  and  “workload”.  Trust  is  unlikely  to  be  reliably  linked  to  performance  and  effectiveness. 
Furthermore,  “trust”  is  unlikely  to  provide  reliable  psychometric  or  concomitant  behavioural  measures 
(qualitative  or  quantitative),  with  useful  sensitivity,  discrimination,  diagnosticity  or  predictive  power. 

7.3. 1.3  Trustworthy  Levels  of  Automation 

For  the  design  of  adaptable  automation  and  automated  decision  support,  research  effort  is  needed  to  be 
directed  at  constructing  and  constraining  human-automation  relationships,  interactions  and  behaviours  in  a 
manner  that  naturally  engender  trust.  Methods  for  delegating  authority  to  automation  are  needed  that  manage 
and  control  risks  of  automation  in  a  sensible,  regulated  and  predictable  manner,  with  appropriate  safeguards. 
Safeguards  are  needed  against  breakdown  or  failure  in  performance  to  ensure  that  operator  trust  in  system 
functioning  is  maintained  at  realistically  appropriate  levels,  without  adversely  affecting  situational  awareness. 
The  first  Law  of  Adaptive  Aiding  states  that  “computers  should  take  tasks,  but  not  give  them”  [19].  Automatic 
reallocation  of  tasks  to  manual  performance  seems  close  to  a  violation  of  this  law.  In  particular,  variable 
assistance  and  allocation  could  lead  to  unacceptable  unpredictability.  So,  awareness  of  the  current  task 
allocation  strategy  is  an  important  factor  for  system  effectiveness,  but  this  may  not  be  easily  achieved  by 
seamlessly  adaptable  aiding.  Careful  consideration  is  needed  of  the  procedures  for  implementation  of  dynamic 
task  allocation  and  re-allocation. 

The  building  of  trust  between  the  operator  and  the  computer  automation  system  has  been  identified  as  a  key 
issue  in  enabling  the  capability  of  cognitive  automation.  Trust  is  built  when  consistency  and  correctness  is 
observed  in  the  computer  system’s  decisions  and  actions.  Two  important  guidelines  for  building  trust  have 
arisen  [23]: 

•  Define  the  Prime  Directives.  These  are  overall  governing  rules  which  bound  the  behaviour  of  the  aiding 
system,  and  yet  provide  a  logical  structure  for  aiding  system  to  act  in  a  rational  and  reliable  manner, 
avoiding  arbitrary  behaviour,  so  that  the  human  does  not  experience  any  surprises,  e.g.,  Asimov’s  Laws 
of  Robotics. 

•  Specify  the  Levels  of  Autonomy.  These  also  bound  the  behaviour  of  the  aiding  system  by  limiting  its 
decision  authority  for  the  performance  of  specific  sub-functions  to  a  set  of  system  configurations 
specified  and  set  by  the  operator. 

Trust  is  built  on  awareness  of  proven  performance.  Adaptive  strategies  for  coping  with  control  of  complex, 
dynamic  situations,  such  as  automatic  unburdening  and  manual  re-allocation,  will  need  careful  adaptive  logic 
to  ensure  their  appropriateness.  The  design  of  the  functional  interface  needs  to  ensure  that  appropriate  levels 
of  awareness  of  the  current  task  allocation  are  easily  maintained.  Awareness  is  needed  to  avoid  task 
contention,  and  to  ensure  that  tasks  are  not  overlooked  or  performed  incorrectly. 

Ideas  of  levels  of  automation  have  been  proposed  to  represent  scales  of  delegation  of  tasks  to  automation,  with 
implications  for  reliability,  use  and  trust.  Sheridan  and  Verplanck  [24]  first  proposed  10  possible  levels  of 
allocation  of  decision-making  tasks  between  humans  and  computers.  More  recently,  Parasuraman,  Sheridan 
and  Wickens  [25]  have  considered  the  application  of  automation  to  a  four-stage  model  of  independent 
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information  processing  functions  (information  acquisition,  analysis,  and  decision  selection  and  action 
implementation).  In  doing  so,  they  have  sought  to  apply  a  revised  set  of  levels  of  automation. 

•  The  computer  decides  everything  and  acts  autonomously,  ignoring  the  human. 

•  The  computer  informs  the  human  only  if  it,  the  computer,  decides  to. 

•  The  computer  informs  the  human  only  if  asked. 

•  The  computer  executes  automatically,  then  necessarily  informs  the  human. 

•  The  computer  allows  the  human  a  restricted  time  before  automatic  execution. 

•  The  computer  executes  the  suggestion  if  the  human  approves. 

•  The  computer  suggests  an  alternative. 

•  The  computer  narrows  the  selection  down  to  a  few. 

•  The  computer  offers  a  complete  set  of  decision  alternatives. 

•  The  computer  offers  no  assistance.  The  human  must  make  all  the  decisions  and  actions. 

The  term  autonomy  has  been  introduced  to  describe  the  bounding  of  functioning  and  decision  authority  of 
advanced  automation  and  intelligent  decision  systems.  Autonomy  can  be  defined  simply  as  the  capability  to 
make  decisions.  Thus,  autonomy  can  be  considered  in  terms  of  the  freedom  to  make  decisions,  considering 
constraints  on  decision-making  (limitations,  boundaries,  rules,  regulations),  decision-making  abilities  (authority, 
responsibility,  competency),  and  the  capability  to  make  different  kinds  of  decisions  (classes,  functions,  levels). 

In  the  1980’s,  the  DARPA/USAF  Pilot’s  Associate  (PA)  program  provided  a  practical  implementation  of 
intelligent  pilot  aiding  based  on  prime  directives  and  levels  of  autonomy  (LOA).  PA  design  was  guided  by  a 
top-level  operational  philosophy  based  on  the  pilot  being  in  charge.  The  goal  of  the  PA  was  to  provide 
consistently  correct  information,  and  to  aid  the  pilot’s  decision  making  by  helping  to  manage  workload, 
reduce  confusion,  and  simplify  tasks.  This  led  to  the  philosophy  of  the  PA  as  an  intelligent  subordinate  to  the 
pilot,  with  specific  capabilities  for  decisions  and  actions.  These  top  level  requirements  led  to  specific 
operational  relationships  (ORs)  for  discrete  PA  sub-functions  interactions,  with  increasing  degrees  of 
automation  and  autonomy.  From  these  ORs,  pilot  selectable  levels  LOA  were  obtained  for  groups  of  functions 
governed  by  the  required  pilot  operational  relationship  and  interaction  [26].  Five  discrete  LOA  modes  were 
proposed,  namely:  Inactive,  Standby,  Advisor,  Assistant,  Associate.  Each  LOA  mode  was  associated  with 
tailorable  functional  clusterings  for  flexible  responding  to  avoid  too  rigid  automation  imposed  by  design. 
These  modes  were  aimed  to  provide  a  bounded,  communicable  structure  for  delegated  levels  of  authority, 
minimising  mode  confusion,  and  building  trust  and  confidence.  Generally,  human  factors  research  indicates 
that  the  required  control  structure  should  be  cognitively  simple,  and  not  complex.  Pilots  tend  to  view 
computer  autonomy  simply  as  either  automatic,  with  or  without  status  feedback;  semi-automatic,  telling  what 
will  happen  and  asking  permission  to  proceed;  or  advisory,  providing  information  only. 

7.3.2  Contractual  Autonomy 

7.3.2.1  Pilot  Authorisation  and  Control  of  Tasks 

In  order  to  address  the  requirement  for  a  practical  implementation  of  LOA,  under  the  UK  MOD  Cognitive 
Cockpit  programme  a  framework  was  created  for  pilot  interaction  and  delegation  of  adjustable  levels  of 
autonomy,  known  as  the  PACT  system  (Pilot  Authorisation  and  Control  of  Tasks).  PACT  spans  the  range  of 
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possible  levels  of  allocation  of  decision  making  tasks,  or  levels  of  autonomy,  between  humans  and  computers 
[24,25].  The  PACT  framework  is  intended  to  provide  trustworthiness  in  the  information  and  behaviour  of 
adaptable  automation  [27,28].  The  PACT  framework  is  summarised  in  Table  7-2. 


Table  7-2:  Bonner-Taylor  PACT  System 


Primary 

Modes 

Levels 

Operational 

Relationship 

Computer 

Autonomy 

Pilot 

Authority 

Adaptation 

Information 

on 

performance 

Automatic 

Automatic 

Full 

Interrupt 

Computer 
monitored  by  pilot 

On/off 

Failure  warnings 
Performance 
only  if  required. 

Assisted 

4 

Direct  Support 

Advised  action 
unless  revoked 

Revoking  action 

Computer  backed 
up  by  pilot 

Feedback  on 
action.  Alerts 
and  warnings  on 
failure  of  action. 

3 

In  Support 

Advice,  and  if 
authorised, 
action 

Acceptance  of 
advice  and 
authorising 
action 

Pilot  backed  up  by 
the  computer 

Feed-forward 
advice  and 
feedback  on 
action.  Alerts 
and  warnings  on 
failure  of 
authorised 
action. 

2 

Advisory 

Advice 

Acceptance  of 
advice 

Pilot  assisted  by 
computer 

Feed-forward 

advice 

1 

At  Call 

Advice  only  if 
requested. 

Full 

Pilot,  assisted  by 
computer  only 
when  requested. 

Feed-forward 
advice,  only  on 
request 

Commanded 

Under  Command 

None 

Full 

Pilot 

None 

performance  is 
transparent. 

PACT  simplifies  the  number  of  automation  modes  required  -  fully  automatic,  assisted,  commanded  -  with  a 
further  secondary  levels  nested  within  the  semi-automatic  assisted  mode,  which  can  be  changed  adaptably  or  by 
pilot  command.  The  PACT  framework  employs  military  terminology  for  categories  of  support  in  British  Army 
land  forces  (Mike  Bonner,  personal  communication).  This  provides  realistic  operational  relationships  compatible 
with  military  user  control  schemata.  It  is  a  logical,  practical  set  of  levels  of  automation,  with  progressive 
operator/pilot  authority  and  computer  autonomy  supporting  situation  assessment,  decision  making  and  action,  as 
illustrated  in  Figure  7-1. 
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Figure  7-1 :  Pact  Progression  of  Operator  Authority  and  Computer  Autonomy. 

The  purposes  of  the  PACT  framework  can  be  summarised  as  follows: 

•  To  bound  the  behaviour  of  the  aiding  system; 

•  To  limit  its  decision  authority  to  the  performance  of  specific  sub-functions;  and 

•  To  enable  a  set  of  system  configurations  to  be  specified,  set  and  adjusted  by  the  operator. 

PACT  is  based  on  the  idea  of  contractual  autonomy.  Using  an  aircrew  term  from  co-operative  air  defence, 
the  pilot  forms  a  set  of  “contracts”  with  the  automation  by  allocating  tasks  to  PACT  modes  and  levels  of 
automation  aiding.  The  contract  defines  the  specific  nature  of  the  operational  relationship  between  the  pilot 
and  the  computer  aiding  for  co-operative  performance  of  specific  sub-functions  and  tasks.  In  setting  the  PACT 
contract,  the  operator  defines: 

•  What  sub-functions  and  tasks  are  aided,  when  and  how; 

•  What  level  of  assistance  is  provided  as  primary  or  default,  and  when; 

•  What  levels  of  assistance  are  permissible  for  anticipatable  contingencies,  and  when; 

•  What  are  permissible  triggers  for  changing  levels  of  assistance,  either  contextual  or  by  operator 
command;  and 

•  What  information  is  provided  to  the  operator,  when  and  how,  including  status  advice,  feed-forward/ 
feedback  course  of  action  information  and  saliency. 

Thus,  autonomy  is  limited  by  the  set  of  contracts  made  between  the  pilot  and  the  computer  automation  system 
governing  and  bounding  the  performance  of  tasks  to  a  set  of  sensible  and  predictable  co-operative  behaviours 
according  to  rules  of  operation  (context,  resources).  Fighter  pilots  develop  similar  inter-personal  contracts  in 
planning  control  of  multi-aircraft  manoeuvres  in  co-operative  air  defence.  The  PACT  autonomy  contracts 
are  binding  delegation  agreements  for  the  computer.  Only  the  pilot  can  set  or  modify  the  PACT  contracts, 
or  define  a  priori  the  contextual  circumstances  for  real-time  adaptation  PACT  changes.  Mission  functions  and 
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tasks  can  be  set  to  PACT  levels  by  allocation  individually  or  grouped  in  related  scripts  or  plays,  at  different 
levels  of  abstraction,  in  a  number  of  ways: 

•  Pre-set  operator  preferred  defaults. 

•  Operator  selection  during  pre-flight  planning. 

•  Changed  by  the  operator  during  in-flight  re-planning,  probably  using  Direct  Voice  Input  commands. 

•  Automatically  changed  according  to  operator  agreed,  context-sensitive  adaptive  rules. 

Figure  7-2  illustrates  a  set  of  mission  functions  and  tasks  with  PACT  contractual  autonomy  levels  arranged 
along  a  timeline  in  a  hypothetical  task  network.  This  provides  the  operator  with  implicit  if  not  explicit  control, 
so  as  to  engender  trust  through  understanding  of  automation  functioning.  Thus,  the  pilot  retains  authority  and 
executive  control,  while  delegating  responsibility  for  the  performance  of  tasks  in  a  sensible  and  predictable 
manner  to  the  computer. 


AUTOMATIC 
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DIRECT 

SUPPORT 
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IN  SUPPORT 
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AT  CALL 


Functions  £  Tasks 
A  Manage  systems  -  Manage  fuel 
B  Make  tactical  decisions  -  Defne  tactical  goal 
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Figure  7-2:  Task  Network  of  Functions  and  Tasks  Set  to  Pact  Contract  Levels. 


In  the  Cognitive  Cockpit  implementation,  the  PACT  system  operates  within  an  adaptive  system  architecture 
that  couples  on-line  monitoring  of  the  pilot’s  functional  state  and  on-line  task  knowledge  management  and 
decision  support  for  context-sensitive  aiding,  deriving  information  to  mediate  the  timing,  saliency  and 
autonomy  of  the  aiding.  The  PACT  framework  provides  the  necessary  and  sufficient  levels  of  autonomy  for 
the  management  of  tasks.  Three  principle  agents  with  different  tasks  comprise  the  Cognitive  Cockpit  system: 

•  A  Cognition  Monitor  (COGMON)  is  responsible  for  monitoring  the  pilot’s  physiology  and  behaviour 
to  provide  an  estimation  of  the  pilot’s  functional  state. 

•  A  Situation  Assessor  Support  System  (SASS)  is  responsible  for  monitoring  the  aircraft  situation  and 
outside  environment,  generating  advice  and  recommending  courses  of  action. 
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•  A  Task  Interface  Manager  (TIM)  is  responsible  for  monitoring  the  mission  plan,  deciding  automation 
and  managing  the  cockpit  interface. 

The  TIM  module  provides  on-line  analysis  of  higher-order  outputs  from  COGMON  and  SASS,  and  other 
aircraft  systems.  A  central  function  for  this  system  is  maximisation  of  the  goodness  of  fit  between  aircraft 
status,  ‘pilot-state’  and  tactical  assessments  provided  by  the  SASS.  These  integrative  functions  enable  this 
system  to  influence  the  prioritisation  of  tasks  and,  at  a  logical  level,  to  determine  the  means  by  which  pilot 
information  is  communicated  through  the  TIM  and  the  associated  cockpit  interfaces.  Overall,  this  system 
allows  pilots  to  manage  their  interaction  with  the  cockpit  automation,  by  context-sensitive  control  over  the 
allocation  of  tasks  to  the  automated  systems. 

The  TIM  functional  architecture  comprised  modules  for  goal-plan  tracking  and  for  interface,  timeline, 
automation  and  task  management  utilising  a  blackboard  for  goal-plan  tracking  information.  Details  of  the  TIM 
functional  architecture  are  provided  elsewhere  [37,38].  The  idea  of  a  tasking  interface  exploits  the  lessons 
learnt  from  the  US  Army’s  RPA  program  [39].  It  arose  from  the  need  to  be  able  to  predict  pilot  expectations 
and  intentions  with  reference  to  embedded  knowledge  of  mission  plans  and  goals.  The  aim  was  to  provide  an 
adaptive  or  “tasking”  interface  that  allowed  the  operators/pilots  to  pose  a  task  for  automation  in  the  same  way 
that  they  would  task  another  skilled  crewmember.  It  afforded  pilots  the  ability  to  retain  executive  control  of 
tasks  whilst  delegating  their  execution  to  the  automation.  A  tasking  interface  necessitated  the  development  of 
a  cockpit  interface  that  allowed  the  pilot  to  change  the  level  of  automation  in  accordance  with  mission 
situation,  pilot  requirements  and/or  pilot  capabilities.  It  was  necessary  that  both  the  pilot  and  the  system 
operated  from  a  shared  task  model,  affording  communication  of  tasking  instructions  in  the  form  of  desired 
goals,  tasks,  partial  plans  or  constraints  that  were  in  accord  with  the  task  structures  defined  in  the  shared  task 
model. 

Providing  flexible  or  adjustable  levels  of  autonomy  for  the  performance  of  tasks  and  functions  is  a  key 
requirement  for  implementation  of  the  tasking  interface  concept.  Allowing  pilots  to  choose  various  levels  of 
interaction  for  the  tasks  they  are  required  to  conduct  can  mitigate  the  problem  of  unpredictability  of 
automation.  TIM  utilises  the  monitoring  and  analysis  of  the  mission  tasks  provided  by  the  SASS  combined 
with  the  pilot  state  monitoring  of  the  COGMON  to  afford  adaptation  of  automation,  adaptable  information 
presentation  and  task  and  timeline  management. 

In  the  Cognitive  Cockpit  implementation,  PACT  levels  are  triggered  adaptively,  in  accordance  with  PACT 
contracts,  in  response  to  contextual  input  from  COGMON,  SASS  and  TIM  mission  goal-plan  tracking  (GPT). 
The  intention  is  to  monitor  and  manage  the  variability  in  performance  through  a  barrier  system  approach 
(monitor,  detect,  correct,  reflect  performance),  and  through  appropriate  cognitive  streaming  interventions 
(join,  break,  divert  cognition).  TIM  feedback  and  feed- forward  control  messages  are  used  with  appropriate 
multi-modal  intervention  saliency  (background,  hinting,  influencing,  directing,  compelling)  developed  to 
reduce  cognitive  bias  with  decision  support  systems.  All  the  tasks  in  the  mission  scenario  are  pre-allocated  to 
possible  PACT  level  contracts  by  the  pilot.  The  individual  task  PACT  levels  (defaults  and  contingencies) 
are  set  to  mitigate  the  risks  to  achievement  of  the  individual  task  goals.  The  TIM  Task  Manager  distinguishes 
between  pending,  active  and  completed  tasks  for  the  current  scenario/vignette.  Individual  tasks  progressed 
from  pending,  to  active,  and  then  to  completed,  as  the  scenario  progressed. 

13.2.2  PACT  Evaluation 

The  operation  of  the  PACT  framework  was  successfully  demonstrated  to  the  MOD  Cognitive  Cockpit  customer 
in  2001.  It  has  subsequently  been  incorporated  into  interfaces  for  UAV  control  and  been  demonstrated 
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successfully  [29].  Diethe  et  al.  [36]  reported  that  a  successful  DARPA  Augmented  Cognition  programme 
Cognitive  Cockpit  closed  loop  trial  was  completed  in  November  2003  during  which  the  stability  and 
performance  of  the  Cognitive  Cockpit  system  were  examined  under  different  levels  of  threat/workload  in  a 
realistic  deep-strike  mission. 

Analysis  has  provided  additional  sources  of  evaluation  information.  A  risk  analysis  [40]  indicated  that  generic 
risks  of  automation  are  likely  to  be  mitigated  by  the  Assisted  PACT  levels,  as  follows: 

•  PACT  5  Automatic:  Automation  bias,  poor  mode  awareness  and  monitoring,  surprise,  unexpected 
action,  ROE  change,  out-of-the-loop  performance,  unpredictability. 

•  PACT  0  Commanded:  Cognitive  bias,  complexity,  pre-occupation,  fixation,  time  pressure,  failure  to 
evaluate  options,  forgetting  rules  and  procedures,  breakdown  of  skill. 

The  PACT  system  is  designed  to  support  the  pilot’s  cognitive  work.  The  support  ranges  from  providing  advice 
to  providing  action.  The  resultant  cognitive  work  can  be  represented  in  terms  of  a  Skills,  Rules,  Knowledge 
(SRK)  perception-assessment-targeting-execution  cognitive  decision  ladder  using  state  flow  transition 
diagrams.  Control  task  analysis  [41,42]  has  been  used  to  identify  the  structure  of  the  cognitive  work 
performed  by  the  pilot  and  by  automation  at  each  PACT  level  [43].  Figure  7-3  illustrates  the  control  task 
analysis  for  PACT  Level  3  Assisted-In  Support,  represented  in  decision  ladder  flow  terms. 


Figure  7-3:  Control  Task  Analysis  for  PACT  Level  3  Assisted  In  Support. 

Figure  7-4  summaries  the  levels  of  cognitive  work  estimated  in  four  decision  ladder  phases  (Perception, 
Assessment,  Decision  and  Action  (PADA).  Workload  estimates  were  provided  for  PACT  levels  with 
immediate  acceptance,  critical  acceptance  and  independent  analysis.  The  analysis  indicated  that  immediate 
acceptance  of  advice,  associated  with  high  levels  of  automation  trust,  was  more  likely  to  occur  for  Perception, 
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Assessment  and  Decision  phases  (i.e.,  situation  assessment,  status,  goals,  options,  effects  and  plans)  - 
but  immediate  authorisation  of  action  was  unlikely  to  occur,  without  critical  appraisal,  indicating  a  basic  lack 
of  trust.  This  may  limit  the  reduction  in  cognitive  load  arising  from  automation  of  advised  action 
(Direct  Support).  Concern  about  the  validity  of  automated  action  is  understandable  during  early 
familiarisation  and  confidence  building.  Critical  appraisal  of  recommended  courses  of  action  probably  will 
continue  until  the  trustworthiness  of  the  system  can  be  established. 
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Figure  7-4:  Cognitive  Load  Estimates  for  PACT  Levels. 


Some  support  for  this  observation  on  the  untrustworthiness  of  action  automation  has  been  reported  in  an 
investigation  of  multiple  UAV  control.  Ruff  et  al,  [44]  describe  a  study  to  assess  the  effects  of  automation 
reliability  and  levels  of  automation  (LOA)  on  supervisory  control  of  multiple  UAVs.  The  LOA  used  were 
Management-by-Consent  (MBC)  and  Management-by-Exception  (MBE).  These  LOA  equate  to  PACT  Levels 
3  and  4  respectively.  Under  MBC,  the  operator  had  to  explicitly  agree  to  suggested  actions  before  they 
occurred.  Under  MBE,  the  system  automatically  implemented  suggested  actions  after  a  pre-set  time  unless  the 
operator  objected.  Results  of  two  experiments  showed  that  participants  reported  higher  workload  and 
difficulty  with  MBE.  Under  MBE  conditions,  time  limits  were  set  on  manual  intervention  before  automatic 
performance  of  the  tasks  (re-plans,  image  prosecution).  This  manipulation  of  reliability  meant  that  erroneous 
action  could  occur.  Participants  generally  chose  to  complete  the  tasks  manually  before  MBE  automatic  action, 
indicating  a  lack  confidence  or  trust  in  the  system  reliability. 
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7.3.3  Supervisory  Control  with  Adjustable  Autonomy 

It  seems  likely  that  future  manned  and  uninhabited  platforms  are  both  likely  to  have  on-board  cognitive 
automation  operating  with  relatively  high  levels  of  autonomy  or  decision  authority.  Context  sensitive 
technologies  or  “intelligent”  computer  software  agents  (e.g.,  Bayesian  nets)  offer  the  possibility  of  being  able 
to  control,  regulate,  direct  and  adapt  system  behaviour,  within  constraint  boundaries,  even  in  uncertain,  novel, 
and  unpredictable  situations.  The  aspiration  is  to  achieve  the  requisite  cognitive  agility,  precision,  reliability 
and  safety  of  operations  with  intelligent  systems,  with  the  minimum  human  supervision  and  human-computer 
communication. 

The  PACT  system  has  been  applied  in  research  on  the  management  of  multiple  UAVs  from  manned  cockpits, 
to  help  reduce  pilot  cognitive  workload.  It  is  seen  as  equally  applicable  to  control  of  multiple  UAVs  from 
ground  stations.  Furthermore,  PACT  seems  particularly  relevant  when  coupled  with  intelligent  organisation 
principles,  control  architectures  and  tools  for  structuring  and  delegating  tasking  co-ordination  and  execution 
workload  [29].  DARPA  sponsored  work  on  air  vehicles  (AV)  indicates  value  in  similar  autonomy  solutions. 
The  DARPA  UCAV  Advanced  Cognition  Aids  Integration  project  for  target  engagement  and  multiple  AV 
identifies  four  levels  of  autonomy,  namely  automate,  exception  (informs  immediate  action,  OK  or  revoke), 
consent  (authorisation  required),  manual  [45].  The  DARPA  ICAV  Intelligent  Control  of  Unmanned  AV 
project  on  mixed  initiative  distributed  intelligence  architecture  for  UAV  operations  identifies  four  levels  of 
authorisation,  namely  autonomous,  veto  (proposal  implicitly  accepted  after  time  out),  permissive  (proposal 
implicitly  rejected  after  time  out),  manual  [46].  For  future  envisioned  UCAV  operations,  involving  real-time, 
multiple  (group)  collaborating  autonomous  vehicles  in  joint  operations  with  manned  platforms,  it  seems  likely 
that  autonomous  control  levels  will  need  extending  beyond  human  command  and  computer  support,  to  cover 
classes  of  autonomous  complimentary,  co-ordinated  and  co-operative  planning  and  interactions. 

Autonomy  issues  and  implementation  solutions  have  been  addressed  in  work  on  multi-agent  intelligent 
systems  for  problem  solving  in  complex  dynamic  environments  [34].  Mixed-initiative  systems,  dynamic 
adaptive  autonomy  and  adjustable  autonomy  have  been  proposed  to  enable  multi-agent  systems  to  perform 
effectively  with  adaptability  and  flexibility.  In  the  context  of  single-agent  to  human-user  interaction, 
autonomy  has  generally  been  viewed  as  freedom  from  human  influence  -  but  for  multi-agent  systems,  where 
the  human  user  may  be  remote  from  operations,  autonomy  becomes  a  matter  of  the  agent’s  self-direction  and 
goals,  and  the  capability  to  dynamically  form,  modify  or  dissolve  the  agent  organisation  into  goal-oriented, 
problem-solving  groups.  The  degree  of  autonomy  is  considered  to  be  implicit  or  explicitly  linked  to  individual 
goals,  and  focuses  on  the  decision  making  process  used  to  determine  how  a  goal  is  pursued  free  from 
intervention,  oversight,  or  control  by  another  agent  (technical  or  human).  Autonomy  with  respect  to  goals  can 
be  considered  to  be  on  a  variable  scale: 

•  Consensus  or  distributed  control  through  consensus  (working  as  a  team  member,  sharing  decision¬ 
making  control  equally  with  all  other  decision-making  agents,  all  with  equal  authority); 

•  Master  control  (makes  decisions  alone,  may  communicate  or  give  orders  to  other  agents  with 
authority); 

•  Locally  autonomous  (makes  decisions  alone,  only  agent  with  authority);  and 

•  Command-driven  or  centralised  control  (makes  no  decisions  about  how  to  pursue  goals,  has  authority, 
but  must  obey  orders  given  by  another  agent). 

Taylor  [32,33]  adds  these  agent  autonomy  levels  to  the  PACT  levels  with  a  summary  of  the  responsibilities  in 
cognitive  control  model  terms  of  advising  and  performing  targeting  (or  governing),  monitoring  (or  directing), 


7-22 


RTO-TR-HFM-078 


dMNATO 
WP  I  OTAN 


HUMAN  AUTOMATION  INTEGRATION 


regulating  and  controlling  (or  operating)  (Table  7-3).  This  enables  consideration  of  the  flow  and  transitioning 
of  control  in  functional  context  rather  than  in  terms  of  internal  decision-making  processes.  Further 
exploitation  of  the  PACT  framework  can  be  suggested  as  follows: 

•  Assign  functions  to  multi-agent  resources  in  CCII.  Use  PACT  levels  to  define  operational  relationships. 

•  Assign  a  broad  range  of  inactive  reserve  functions  and  operational  relationships  to  PACT  Level  1 
Assisted  At  Call,  i.e.,  pre-set  at  PACT  Levels  2,  3,  4. 

•  Use  PACT  to  define  multi-agent  support  inter-relationships  at  the  Master  Control  autonomy  level. 

•  Use  PACT  agents  to  organise  and  filter  prioritised  information  in  Command  and  Control  Information 
Infrastructure  (CCII)  for  command  intent  and  SA. 

Table  7-3:  Adjustable  Autonomy  Levels  for  Intelligent  Multi-Agent  Systems 


AUTONOMY 

TARGETING 

(GOVERNING) 

MONITORING 

(DIRECTING) 

REGULATING 

CONTROLLING 

(OPERATING) 

Consensus 

Autonomy 

Multiple  intelligent 
computer  agents 

Multiple  intelligent 
computer  agents 

Multiple  intelligent  computer 
agents 

Multiple  intelligent 
computer  agents 

Master 

Autonomy 

Intelligent  computer  agent 

Intelligent  computer 
agent 

Intelligent  computer  agent  + 
Authorised  support  agents 

Intelligent  computer  agent  + 
Authorised  support  agents 

Local 

Autonomy 

Intelligent  computer  agent 

Intelligent  computer 
agent 

Intelligent  computer  agent 

Intelligent  computer  agent 

Automatic/ 

Commanded 

Autonomy 

Operator 

Computer  agent 
performing  some 
interpretation  &  planning 
+  Operator  interrupt 

Computer  agent  performing 
recognition  &  scheduling  + 
Operator  interrupt 

Computer/ intelligent  agent 
performing  detection  & 
execution  agent  +  Operator 
interrupt 

Assisted  Direct 
Support 

Operator 

Operator  authorising  + 
Computer  agent 
performing  some 
interpretation  &  planning 

Operator  authorising  + 
Computer  agent  performing 
recognition,  &  scheduling 

Operator  authorising  + 
Computer  agent  performing 
detection  &  execution 

Assisted  In 
Support 

Operator 

Operator  performing  + 
Optional  computer  agent 
advising  &  performing 
some  interpretation  & 
planning 

Operator  performing  +  Optional 
computer  agent  advising  & 
performing  recognition  & 
scheduling 

Operator  performing  + 
Optional  computer  agent 
advising  &  performing 
detection  &  execution 

Assisted 

Advisory 

Operator 

Operator  performing  + 
Computer  agent  advising 
interpretation  &  planning 

Operator  performing  + 
Computer  agent  advising 
recognition  &  scheduling 

Operator  performing  + 
Computer  agent  advising 
detection  &  execution 

Assisted 

At  Call 

Operator 

Operator  +  Optional 
computer  agent 

Operator  +  Optional  computer 
agent 

Operator  +  Optional 
computer  agent 

Command 

Operator 

Operator 

Operator 

Operator 

Adjustable  autonomy  gives  the  agent  architecture  the  ability  to  adapt  their  problem-solving  to  situations 
particularly  in  domains  with  unreliable  communications  and  the  possibility  of  agent  failure,  high  degrees  of 
uncertainty  and  resource  contention  needing  distribution  of  tasks  and  co-ordinated  planning  to  resolve 
conflicts.  Distributed  problem  solving  structures  are  generally  thought  to  perform  faster  for  complex  tasks, 
when  operating  under  uncertainty  and  changes  in  the  environment,  when  few  resources  are  shared,  and  when 
communication  is  unreliable.  Centralised  structures  perform  faster  for  simple  tasks,  when  many  resources  are 
shared,  when  communication  is  reliable,  and  when  there  is  no  requirement  to  negotiate.  Autonomy  level 
agreements  and  communication  protocols,  joint  intentions,  and  employing  conventions  for  explicit 
commitment  to  specific  interaction  styles  are  considered  necessary  to  establish  reliability  and  trust.  A  central 
problem  in  adjustable  autonomy  is  the  determination  of  whether  and  when  transfers  of  control  to  the  operator/ 
user  should  occur  [47].  The  transfer  of  control  from  agent  to  human  is  believed  to  require  a  balancing  of  the 
costs  of  interrupting  a  human  user  with  the  benefits  for  highest  quality  decision  making  when  the  human  has 
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superior  decision-making  expertise.  One  technique  proposes  that  transfer  should  occur  when  the  expected 
utility  of  transfer  is  greater  than  that  of  retaining  the  decision-making.  Another  forces  the  agent  to  relinquish 
and  transfer  control  if  the  uncertainty  is  high.  Others  transfer  if  any  incorrectness  in  the  agents  decision  can 
cause  significant  harm,  if  the  agent  lacks  decision-making  capability,  or  on  the  basis  of  thresholds  of  learnt 
rules.  In  multi-agent  applications,  cognitive  strategies  are  needed  for  reasoning  with  adjustable  autonomy  in 
the  operating  context  (situated  autonomy)  to  provide  the  correct  co-ordination,  reordering  and  scheduling  and 
to  balance  the  costs,  benefits,  uncertainty  and  implications  within  the  multi-agent  group  [48]. 

There  is  considerable  potential  for  read-across  for  control  architectures  from  cognition  and  joint  cognitive 
systems  for  the  control  of  distributed  multi-agent  systems.  They  use  decision  resources  efficiency  and  enable 
the  decision  agility  and  adaptiveness  needed  for  the  manouevrist  approach  to  military  problem-solving. 
The  use  of  cognitive  control  models  will  increase  the  transparency  of  control  architectures  and  control 
authority  for  human  user  appreciation  of  the  planning  and  interaction  situation  during  collaborative  problem¬ 
solving. 


7.3.4  Conclusions 

The  PACT  framework  developed  for  pilot  authorisation  of  control  of  tasks  provides  a  simplified,  practical  set 
of  adjustable  levels  of  contractual  autonomy  capable  of  engendering  trust  in  automation.  An  illustration  of  the 
idea  of  PACT  as  an  enabling  framework  between  and  automation  trust,  reliability  and  use  is  shown  in 
Figure  7-5.  PACT  enables  the  pilot  to  delegate  responsibility  for  tasks  to  the  computer  through  a  set  of 
contracts  that  limit  autonomy  and  bound  the  behaviour  of  the  aiding  system,  while  maintaining  the  pilot’s 
authority  through  executive  control.  Control  task  analysis,  cognitive  loading  and  risk  analysis  provide  useful 
tools  for  understanding  and  modelling  the  functioning  of  the  PACT  system.  The  PACT  framework  seems 
sufficiently  robust  and  useful  to  be  applicable  to  other  systems  and  environments  requiring  cognitive  control 
with  trustworthy  variable  levels  of  autonomy,  such  as  the  control  of  multiple  uninhabited  vehicles. 


A  number  of  fundamental  questions  and  key  issues  can  be  identified  concerning  the  role  of  humans  in 
advanced  automated  and  intelligent  systems.  In  particular,  there  is  uncertainty  over  how  to  optimise  the  use  of 
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human  and  computer  decision  resources,  while  preserving  a  human-centric  system.  These  matters  need  to  be 
understood  in  the  context  of  the  changing  capability  requirement  responding  to  new  military  problem-solving 
challenges.  Important  changes  are  being  made  in  the  way  in  which  military  force  is  to  be  used  in  the  future 
with  the  introduction  of  effects-based  approach  to  the  planning  and  conduct  of  joint  operations.  This  will  be 
enabled  by  network  CCII,  and  will  provide  shared  planning  and  situation  appreciation,  command  intent  and 
Combat  Identification.  The  prime  reason  for  human  involvement  in  military  decision-making  with  automated 
systems  -  human  control  of  use  of  military  force  for  safety  assurance  -  seems  established  in  military  law  and 
it  is  axiomatic  for  military  relevance.  Human  knowledge,  experience  and  judgement  provide  unique  capability 
to  analyse  safety  risks  and  to  think  ahead  in  uncertain  and  novel  situations.  The  challenge  is  to  provide 
information  and  decision  systems  that  protect  and  preserve  the  human  user’s  key  role,  and  that  augment  and 
enhance  the  user’s  cognition  rather  than  replaces  the  user  in  complex  decision  making.  Recent  developments 
in  theory  of  cognition  provide  pragmatic  approaches  that  are  likely  to  improve  understanding  of  the  human 
factors  issues,  problems  and  solutions  of  human-computer  collaboration.  In  addition,  new  approaches  to  the 
use  of  automation  propose  adjustable  levels  of  computer  autonomy  with  a  strong  socio-technical  and 
cognition  basis.  These  seem  likely  to  provide  sensible  architectures  for  distributed,  multi-agent  intelligent 
systems  that  can  be  more  readily  appreciated  by  human  users  than  traditional  automation  approaches. 
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7.4  ADAPTIVE  AUTOMATION  FOR  ROBOTIC  MILITARY  SYSTEMS 
7.4.1  Introduction 

Future  Combat  Systems  (FCS)  is  a  US  Army  program  that  will  transform  the  battlefield  Future  force 
structure;  doctrine  and  tactics  will  change  as  new  systems  are  introduced;  possibly  in  ways  that  can  not  be 
anticipated.  Units  of  Action  (UA)  are  being  designed  to  be  flexible,  reconflgurable  components  of  FCS 
tailored  to  specific  combat  missions.  One  aspect  of  increased  flexibility  will  be  the  introduction  of  numerous 
robotic  systems.  The  term  robot  is  used  in  a  generic  sense  to  describe  systems  that  are  unmanned  with  some 
degree  of  autonomy  that  include  aerial,  ground,  subterranean,  naval  surface  and  sub-surface  vehicles.  These 
systems  will  be  an  essential  part  of  the  future  force  because  they  extend  manned  capabilities,  are  force 
multipliers  and  most  important,  they  can  save  lives. 

Any  major  change  in  current  doctrine  implies  problems  as  well  as  solutions.  Robotic  systems  with  diverse 
roles,  tasks  and  operating  requirements  are  being  designed  to  exploit  future  battle  spaces.  The  role  of  the 
human  operator  is  not  well  understood;  however  most  of  the  contemplated  systems  will  require  either  active 
human  control  or  supervision  with  the  possibility  of  intervention.  In  the  most  extreme  case,  soldiers  will 
operate  multiple  systems  while  on  the  move  and  while  under  enemy  fire.  In  all  cases,  the  workload  and  stress 
will  be  variable  and  unpredictable  -  changing  rapidly  as  a  function  of  the  military  environment.  The  purpose 
of  this  chapter  is  to  investigate  technologies  that  unload  the  warfighter  interacting  with  unmanned  systems 
during  multi-tasking  missions.  First,  we  will  investigate  automation  technologies,  specifically  their  positive 
and  negative  effects  on  human  performance  and  situation  awareness.  Next,  we  will  discuss  adaptive  and 
adaptable  processes  as  methods  that  potentially  overcome  the  disadvantages  of  preset  automation.  The  last 
section  will  survey  diverse  physiological  measures  that  can  be  used  to  trigger  adaptive  processes  emphasizing 
the  rapid  development  of  these  methods  and  their  current  limitations. 

Future  robotic  systems  are  being  designed  to  be  used  in  all  facets  of  the  modern  battlespace  and  be,  to  the 
degree  possible,  autonomous.  This  requires  both  rapid  response  capabilities  and  intelligence  built  into  the 
system.  However,  ultimate  responsibility  for  system  outcomes  always  resides  with  the  human  and  in  practice; 
even  highly  automated  systems  usually  have  some  degree  of  human  supervisory  control  [1].  Particularly  in 
combat,  some  oversight  and  the  capability  to  override  and  control  lethal  systems  will  always  be  a  human 
responsibility  for  the  following  reasons  [2]: 

•  System  safety; 

•  Change  in  the  commander’s  goals; 

•  Implied  meta-goals;  and 

•  Fratricide. 
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ORGANIZATION 


However,  automation  is  not  an  all  or  nothing  phenomenon.  Automation  can  vary  in  the  degree  to  which  a 
particular  function  that  was  previously  carried  out  by  a  human  operator  is  allocated  to  a  machine  agent.  This  is 
the  concept  of  “level  of  automation”  (LOA)  as  discussed  by  Sheridan  [3].  However,  automation  can  vary  in 
other  dimensions  as  well,  for  example  in  the  stage  of  human  information  processing  that  the  automation  is 
applied,  whether  to  stages  such  as  information  acquisition  and  analysis,  or  to  stages  such  as  decision  making 
and  response  execution.  Parasuraman,  Sheridan  and  Wickens  [4]  developed  a  taxonomy  of  human  automation 
control  that  is  two  dimensional.  Figure  7-6  shows  rows  as  degree  of  automation  and  columns  as  type  of 
processing  function  addressed.  The  four  processing  functions  of  information  acquisition,  information  analysis, 
decision  making/action  selection,  and  action  implementation  are  similar  to  the  Observe-Orient-Decide-Act  or 
OODA  loop  in  the  parlance  of  military  command  and  control.  The  taxonomy  also  captures  the  multiplicity  of 
control  options  from  fully  automated  to  fully  manual  for  each  of  these  functions.  The  decision  space  is  not 
only  complex,  but  it  implies  that  there  is  not  a  single  solution  to  partitioning  control.  Specifically,  as  the  type 
of  task  the  operator  performs  changes  the  control  logic  may  need  to  change  as  well.  This  is  understating  the 
problem  because  the  taxonomy  does  not  consider  either  other  tasks  the  operator  is  performing  or  the  overall 
workload  and  stress  imposed  by  the  current  environment.  A  review  of  the  human  performance  literature 
reinforces  the  notion  that  there  is  not  a  single  solution  to  partitioning;  human  performance  varies  greatly 
depending  on  the  operator  task  and  the  current  environment  [5,4]. 


Processing  task- 
type  of  auto 

Info 

acquisition 

Info 

analysis 

Action 

selection 

Action 

implementation 

Full  auto 

Auto-human 

informed 

sometimes 

Auto-human 

Informed 

Auto-human  veto 
time  limit 

Auto  executes  only 
if  human  approves 

Auto  suggests 

Auto  narrows 

Auto  shows  all 
options 

Manual 

Figure  7-6:  Human-Automation  Taxonomy  with  Rows  Representing  Degree  of 
Automation  and  Columns  Processing  Functions.  (Adapted  from  [4]). 


7.4.2  Human  Performance  Issues  for  Automated  Systems 

Numerous  problems  related  to  human  performance  in  automated  systems  have  been  identified  in  the  literature. 
One  problem  with  automated  systems  is  the  operator’s  trust  and  level  of  use  of  the  automation.  Parasuraman 
and  Riley  [5]  compiled  both  research  and  real  world  examples  of  automation  misuse,  disuse  and  abuse. 
They  showed  that  the  human  operator  ignored  important  indicators,  failed  to  use  reliable  systems,  misused 
unreliable  systems,  or  misunderstood  the  true  state  of  the  system.  The  paper  catalogued  various  human 
deficiencies  related  to  supervisory  control  and,  most  important;  the  review  motivated  a  good  deal  of 
subsequent  research. 
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In  another  review  article,  Mosier  and  Skitka  [6]  examined  what  they  termed  cases  of  automation  bias. 
The  operator  tended  to  over  rely  on  automated  systems  even  in  cases  where  appropriate  operator  intervention 
would  have  averted  performance  problems.  They  did  not  identify  an  automation  bias  per  se,  but  rather 
identified  a  number  of  performance  problems  related  to  lowered  vigilance,  high  workload,  time  stress  and  loss 
of  situation  awareness  (SA).  SA  problems  resulted  from  a  combination  of  automation  complexity  and  poor 
display  design.  The  best  example  was  the  Three  Mile  Island  accident  wherein  operators  were  misled  by  too 
many  malfunction  indicators  that  were  unrelated  to  the  underlying  problem.  Again,  there  did  not  seem  to  be  a 
coherent  theory  explaining  automation  bias  rather  the  authors  identified  multiple  causes  of  over  reliance. 
Other  researchers  found  that  while  there  were  circumstances  where  humans  over  relied  on  automation,  there 
were  other  equally  important  instances  where  they  should  have  relied  on  automation  and  did  not. 

For  example,  an  Army  supported  study  investigated  operator  trust  issues  related  to  the  Battlefield  Combat 
Identification  System  (BCIS)  [7].  In  an  experiment  with  college  students,  the  simulated  aid’s  target 
identification  rate  was  varied  from  60  -  90%  accuracy  and  the  participants’  task  was  to  affirm  or  override  the 
target  decisions.  Overall  the  subjects  were  twice  as  likely  to  be  wrong  when  they  agreed  with  an  erroneous  aid 
decision  (p  (error/aid  error)  =  .27)  then  when  they  made  override  errors  for  cases  when  the  aid  was  correct 
(p  (error/aid  correct)  =  .13)  supporting  the  automation  bias  hypothesis. 

However,  other  research  from  the  same  authors  indicated  exactly  the  opposite  bias-  disuse  of  appropriate 
automation.  In  a  study  similar  to  the  BCIS  study,  college  students  were  given  200  trials  in  which  they  had  to 
decide  whether  a  target  was  present  on  the  display  or  not  [8].  After  each  trial,  target  advisories  were  given  to 
the  students  purported  to  be  from  either  an  aid  (automatic  target  recognition  (ATR)  device)  or  a  “peer”. 
Participants  were  told  the  relative  accuracy  rates  for  the  aids  (“peers”)  and  their  own  decisions  and  then  had  to 
decide  whether  to  base  future  decisions  on  their  own  performance  or  that  of  the  automated  advisory  (or  peer). 
Surprisingly,  even  when  the  aid  made  Vi  as  many  errors  and  the  subjects  reward  depended  on  accuracy,  80% 
of  the  subjects  chose  to  make  their  own  decisions.  Also,  subjects  trusted  “peer”  advisories  more  than  the  aid 
advisory  with  the  same  accuracy  level.  They  rationalized  their  decisions  in  terms  of  self  reliance.  However, 
the  most  salient  difference  between  this  study  and  the  BCIS  study  was  that  subjects  were  told  the  ATR-  “peer” 
decision  after  they  had  made  their  own  decision.  Thus  there  was  no  workload  advantage  to  using  the  aid 
advisory  because  the  operator’s  decision  was  made  before  the  aids  results  were  known. 

The  results  are  important  because  a  simple  manipulation  (the  order  in  which  decisions  were  required)  caused 
an  automation  bias  to  shift  to  a  self-reliance  bias.  The  same  results  were  repeated  in  subsequent  experiments, 
but  the  self-reliance  bias  was  mitigated  if  the  test  subject  was  informed  the  reason  for  the  ATR  errors  and 
were  given  appropriate  feedback  during  the  initial  trials  [9].  Anecdotal  data  also  indicated  mistrust  of  aids  in 
cases  where  there  was  a  high  false  alarm  rates:  the  “cry  wolf’  phenomenon  [10].  In  summary,  humans  are 
neither  universally  over  reliant  or  under  reliant  on  automated  systems.  The  crucial  factors  seem  to  be 
workload,  time  stress,  false  alarm  rate  and  decision  order. 

Automation  reliability  has  the  same  contradictory  effects  on  performance  depending  on  the  task,  workload  and 
type  of  errors  the  automated  device  makes.  A  number  of  experimenters  found  no  effect  of  aid  reliability  on 
performance  [7,11,12].  The  most  likely  reason  is  the  lack  of  calibration  of  the  subjects.  For  example,  in  the 
Dzindolet  et  al.  study  each  subject  received  only  one  reliability  level  (60%,  75%  and  90%)  and  they  were  not 
given  the  feedback  that  may  have  allowed  them  to  respond  effectively.  The  literature  suggests  that  humans 
have  problems  understanding  probabilities  and  may  need  some  form  of  intervention  in  order  to  perform 
efficiently  [13,14].  Reliability  level  effects  depend  both  on  operator  strategies  and  the  type  of  error  the  aid 
manifests.  Meyer  [15]  has  shown  that  when  automation  reliability  is  such  that  malfunctions  are  almost  always 
correctly  indicated  -  that  is,  the  automation  makes  few  misses,  then  the  operator  has  high  reliance  on  the 
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automation.  This  is  an  effective  strategy,  but  can  result  in  a  problem  when  the  automation  does  miss,  because 
of  the  complacency  effect.  On  the  other  hand,  if  automation  reliability  is  such  that  few  false  alarms  are  made, 
then  the  operator  has  high  compliance :  if  an  automated  alarm  sounds,  then  the  operator  tends  to  immediately 
comply  with  the  alarm  and  tend  to  the  situation.  Reliance  on  automated  aids  permits  the  operator  to  attend  to 
tasks  other  than  the  automated  task  until  the  alert  is  triggered  thus  improving  multitask  performance  and  not 
just  the  performance  on  the  automated  task. 

Research  using  realistic  unmanned  aerial  vehicle  (UAV)  operator  tasks  indicted  that  reliant  behaviors  are 
affected  principally  by  the  misses.  In  contrast,  compliance  errors  were  affected  by  the  false  alarm  rate,  but  not 
affected  by  miss  rate  of  the  automated  device.  However,  increasing  the  operator’s  workload  and  decreasing 
the  aid’s  reliability  level  had  adverse  effects  on  both  compliance  and  reliance  errors  [16]. 

The  issue  is  complicated  because  performance  depends  on  the  type  of  processing  task  the  operator  performs 
and  paradoxically  high  reliability  can  result  in  costs  as  well  as  benefits.  Rovira,  McGarry  and  Parasuraman 
[17]  investigated  automation  of  artillery  targeting  decisions.  For  their  particular  task,  they  found  that  reliable 
automation  improved  the  surrogate  commander’s  decision  latency  without  sacrificing  accuracy.  However, 
particularly  for  decision  tasks  related  to  choosing  a  course  of  action,  higher  reliability  hurt  the  operator’s 
performance  when  the  aids  gave  incorrect  information.  The  surrogate  commanders  trusted  80%  accurate  aids 
more  than  the  60%  ones  in  cases  where  they  should  have  been  more  skeptical  (i.e.,  when  the  aids  gave  them 
incorrect  information).  Apparently,  the  advisories  from  Trusted”  aids  were  not  scrutinized  as  thoroughly  as 
those  from  less  reliable  aids. 

One  interpretation  of  the  previous  results  is  to  assume  that  reliable  aids  lulled  the  operator  into  a  false  sense  of 
complacency.  However,  the  complacency  literature  suggests  that  the  results  depend  on  other  factors  besides 
trust.  A  number  of  researchers  had  failed  to  show  complacency  effects,  motivating  Parasuraman,  Molloy  and 
Singh  [11]  to  investigate  possible  reason  for  this  in  a  multitask  aviation  environment.  There  most  important 
finding  was  that  complacency  did  not  occur  for  low  workload  (single  task)  conditions.  For  the  high  workload 
task,  the  operator  became  complacent  (over  relied)  on  the  aid  when  the  aid  had  a  constant  reliability  level; 
however,  when  the  reliability  level  varied  over  a  block  of  trials  the  performance  decrement  was  ameliorated. 
This  suggests  that  complacency  was  not  so  much  a  matter  of  trust,  but  a  strategy  to  deal  with  high  workload. 
If  an  aid  acted  in  a  predictable  manner  (constant  reliability)  then  the  operators  would  commit  their  resources 
to  other  tasks  in  a  high  workload  environment  causing  a  performance  decrement  in  the  unmonitored 
automation  task. 

The  forcing  function  in  most  of  these  studies  was  high  workload.  The  operator  basically  traded  off  situation 
awareness  for  workload  reduction  by  depending  on  aids  even  in  cases  when  it  was  not  beneficial  to  do  so. 
This  was  not  universally  true,  in  some  cases  automation  improved  overall  performance  even  when  the 
automated  task  required  intervention  because  the  operator’s  residual  cognitive  capacity  was  allocated 
effectively  among  the  set  of  tasks  [18,19].  However,  too  often,  the  loss  of  situation  awareness  related  to 
inefficient  automation  monitoring  leads  not  only  to  performance  decrements,  but  also  to  an  increasingly 
impoverished  understanding  of  the  work  environment  which  can  over  time  result  in  catastrophic  errors 
[20,6,5].  Because  of  the  uncertainty  and  risk  associated  with  military  environments,  automating  any  but  the 
most  trivial  tasks  must  be  done  with  extreme  caution.  The  soldier  and  his  command  chain  need  to  maintain 
situation  awareness;  keeping  the  soldiers  out  of  the  loop  will  not  only  have  consequences  for  the  immediate 
task,  but  also  predispose  them  to  miss  important  cues  signaling  change  [20].  Conversely,  requiring  soldiers  to 
engage  in  multiple  tasks  could  very  well  have  the  same  consequences. 
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7.4.2.1  Adaptive  Principles 

A  possible  solution  is  to  create  enough  flexibility  in  the  system  to  ensure  more  automation  during  peak 
workload  and  greater  operator  engagement  during  workload  lulls.  However,  workload  itself  is  a  theoretical 
construct  and  it  is  not  always  obvious  how  it  affects  performance.  For  example,  the  traditional  view  of 
vigilance  was  that  underload  was  responsible  for  the  observed  human  performance  decrement  as  a  function  of 
time  of  watch.  Recent  research  indicates  that  it  is  overload  and  not  underload  that  is  responsible  for  the 
deleterious  effects  of  vigilance  [21].  A  possible  work  around  would  be  to  leave  it  up  to  the  operator  to  decide 
when  to  automate  and  which  tasks  to  automate  as  mission  requirements  change.  However,  this  is  not  always 
practical  because  it  would  burden  operators  with  additional  tasks  precisely  when  they  are  already  heavily 
loaded.  For  this  reason,  a  number  of  researcher  have  suggested  using  some  form  of  behavioral  indictor  to 
change  levels  of  automation  dynamically  as  a  function  of  the  changing  work  environment  [22-25]. 

Adaptive  automation  uses  mitigation  criteria  that  drive  an  invocation  mechanism  to  maintain  an  effective 
mixture  of  operator  engagement  and  automation  for  a  dynamic  multitask  environment  (Figure  7-7). 
The  invocation  mechanism  is  triggered  by  whatever  measurement  process  is  used  to  represent  the  current  task 
state.  If  properly  instrumented  the  results  of  the  measurement  process  should  be  displayed  to  operators  in 
order  to  keep  them  informed  of  the  state  of  the  invocation  process. 


Figure  7-7:  Example  of  a  Closed  Loop  Adaptation  for  A  -  Automated, 
A/M  -  Automated/Manual,  and  M  -  Manual  Task  Sets. 


This  construct  is  more  complex  than  simply  unloading  (or  engaging)  the  operator  because  to  be  effective  the 
invocation  process  must  be  sensitive  to  the  operator’s  combined  tasking  environment  which  depends  on 
interactions  among  tasks  as  well  as  overall  workload,  stress  and  safety  considerations  [26].  For  example, 
the  algorithm  might  automate  auditory  tasks  when  the  communication  traffic  reaches  a  predefined  level, 
but  not  change  other  task  states  until  the  overall  workload  measure  (or  physiological  index)  reaches  criterion 
[27].  Furthermore,  whenever  certain  critical  events  occur,  the  invocation  mechanism  must  be  sensitive  to  indices 
that  imply  that  the  operator  requires  emergency  automation  (e.g.,  G-loss  of  consciousness  as  evidenced  by 
physiological  measurements)  [28]. 
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7.4.3  Some  Characteristics  of  Adaptive  Automation  Systems 

7.4.3. 1  Invocation  Methods 

In  adaptive  systems,  the  “division  of  labor”  between  human  and  machine  agents  is  not  fixed  but  dynamic, 
in  contrast,  to  systems  where  provision  of  computer  aiding  is  pre-determined  at  the  design  stage,  and  task 
allocation  is  fixed  during  system  operations.  Although  the  adaptive  automation  concept  is  not  new,  having 
been  proposed  about  25  years  ago  [29],  technologies  needed  for  its  effective  implementation  were  not  readily 
available  until  recently.).  A  key  issue  in  adaptive  automation  is  the  method  of  invocation.  Parasuraman  et  al. 
[23]  reviewed  the  major  techniques  and  found  that  they  fell  into  five  main  categories: 

•  Critical  events; 

•  Operator  performance  measurement; 

•  Operator  physiological  assessment; 

•  Operator  modelling;  and 

•  Hybrid  methods. 

The  critical-events  method  is  exemplified  by  the  work  of  Barnes  and  Grossman  [28].  In  this  approach, 
automation  is  invoked  when  certain  tactical  environmental  events  occur,  but  not  otherwise.  For  example,  in  an 
aircraft  air  defence  system,  the  beginning  of  a  “pop-up”  weapon  delivery  sequence  leads  to  the  automation  of 
all  defensive  measures  of  the  aircraft.  If  the  critical  events  do  not  occur,  the  automation  is  not  invoked.  Hence 
this  method  is  inherently  flexible  and  adaptive,  because  it  can  be  tied  to  current  tactics  and  doctrine  during 
mission  planning. 

However,  a  disadvantage  of  the  method  is  its  possible  insensitivity  to  actual  system  and  human  operator 
performance.  For  example,  this  method  will  invoke  automation  irrespective  of  whether  or  not  the  pilot 
requires  it  when  the  critical  event  occurs.  Operator  performance  and  physiological  measurement  attempts  to 
overcome  this  limitation.  In  these  methods,  various  operator  mental  states  (e.g.,  mental  workload,  or  more 
ambitiously,  operator  intentions)  may  be  inferred  on  the  basis  of  performance  or  other  measures  and  then 
input  to  adaptive  logic.  For  example,  performance  and  physiological  measurements  may  allow  the  inference 
that  a  human  operator  is  dangerously  fatigued  or  experiencing  extremely  high  workload.  An  adaptive  system 
could  use  these  measurements  to  provide  computer  support  or  advice  to  the  operator  that  would  mitigate  the 
potential  danger.  Alternatively,  human  operator  states  and  performance  may  be  modeled  theoretically, 
with  the  adaptive  algorithm  being  driven  by  the  model  parameters.  Intelligent  systems  that  incorporate  human 
intent  inferencing  models  have  been  proposed  [30].  Finally,  hybrid  methods  could  be  used  that  combine  one 
or  more  of  these  different  invocation  techniques,  so  that  their  relative  merits  can  be  maximized. 

7.4.3.2  Adaptive  and  Adaptable  Systems 

In  adaptive  systems,  the  decision  to  invoke  automation  or  to  return  an  automated  task  to  the  human  operator  is 
made  by  the  system,  using  any  of  the  previously  described  invocation  methods.  This  immediately  raises  the  issue 
of  user  acceptance  of  such  a  system.  Human  operators  may  be  unwilling  to  accede  to  the  “authority”  of  a 
computer  system  that  mandates  when  and  what  type  of  automation  is  or  is  not  to  be  used.  Apart  from  user 
acceptance,  however,  is  the  issue  of  system  unpredictability  and  its  consequences  for  operator  performance. 
Billings  and  Woods  [31],  for  example,  raised  the  caution  that  truly  adaptive  systems  may  be  problematic  because 
their  behavior  may  not  be  predictable  to  the  user.  To  the  extent  that  automation  can  hinder  the  operator’s 
situation  awareness  by  taking  him  or  her  out  of  the  loop,  unpredictably  invoked  automation  by  an  adaptive 
system  may  further  impair  the  user’s  SA.  However,  if  the  automation  were  explicitly  invoked  by  the  user, 
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then  presumably  the  unpredictability  will  be  lessened  -  but  involving  the  human  operator  in  making  decisions 
about  when  and  what  to  automate  can  increase  workload.  Thus,  there  is  a  tradeoff  between  increased 
unpredictability  versus  increased  workload  in  systems  in  which  automation  is  invoked  by  the  system  or  by  the 
user,  respectively.  Opperman  [32]  characterized  these  alternatives  as  ‘adaptive’  and  ‘adaptable’  approaches  to 
system  design  (see  also  [33]).  In  either  case,  the  human  +  machine  system  adapt  to  various  contexts,  but  in 
adaptive  systems  automation  determines  and  executes  the  necessary  adaptations,  whereas  in  adaptable  systems, 
the  operator  is  in  charge  of  the  desired  adaptations.  The  distinction  is  primarily  one  of  authority.  In  an  adaptable 
system,  the  human  always  maintains  authority  to  invoke  or  change  the  automation,  whereas  this  authority  is 
shared  in  an  adaptive  system.  Inagaki’s  [34]  design  concept  of  “situation-adaptive  autonomy”  is  related  to  this 
view  of  an  adaptive  system,  but  in  his  approach,  control  of  a  process  is  traded  off  between  human  and  computer 
in  real  time  based  on  time  criticality  and  the  expected  costs  of  human  and  machine  performance. 

While  in  this  review  we  primarily  consider  how  adaptive  automation  affects  system  performance,  it  is 
important  to  keep  in  mind  that  adaptable  automation  may  provide  an  alternative  approach  with  its  own 
benefits.  The  LOA  concept  introduced  by  Sheridan  [3]  does  not  specify  which  level  should  be  used  or  who 
decides  that  there  should  be  a  change  in  level. 

When  the  decision  is  made  by  a  designer  prior  to  system  operation,  it  is  a  part  of  system  design  and 
corresponds  to  picking  an  appropriate  LOA  for  that  system  design.  The  decision  can  also  be  made  by 
automation  itself  (or  some  expert  system)  during  system  operators,  as  a  part  of  a  truly  adaptive  automation 
system.  In  both  of  these  cases,  the  human  operator  is  not  involved  in  the  decision.  In  adaptable  systems, 
however,  the  human  operator  is  more  akin  to  a  supervisor  of  a  human  team  who  delegates  tasks  to  team 
members,  or  in  this  case,  to  automation.  The  challenge  for  developing  such  an  adaptable  automation  system  is 
that  the  operator  should  be  able  to  make  decisions  regarding  the  use  of  automation  in  a  way  that  does  that 
create  such  high  workload  that  any  potential  benefits  of  delegation  are  lost. 

One  such  architecture  for  adaptable  automation  that  can  provide  for  flexible  tasking  of  automation  is  the 
“Playbook”  [35-37].  The  objective  of  the  Playbook  interface  is  to  provide  a  human  supervisor  the  ability  to 
delegate  tasks  to  automation  with  much  of  the  flexibility  available  in  human-human  task  delegation,  and  to  do 
so  dynamically  at  the  time  of  system  operation  rather  than  at  system  design.  The  Playbook  interface  facilitates 
the  “teaching”  of  automation  (an  idea  first  proposed  by  Sheridan  [3]  when  he  created  the  supervisory  control 
concept)  by  creating  a  shared  knowledge  structure  of  tasks  and  their  relationships  within  which  task 
performance  can  be  discussed  by  human  and  automation.  For  the  Playbook  concept  to  work,  the  automation 
must  have  substantial  knowledge  about  how  to  perform  tasks  and  achieve  goals.  This  knowledge  is  also  used 
to  improve  the  efficiency  and/or  safety  of  plans  developed  by  allowing  automation  to  review  and  critique 
human  plans.  Finally,  Playbooks  streamline  the  process  of  delegation  by  the  human  operator  by  providing  a 
compiled  set  of  plans,  or  ‘plays’,  with  short,  easily-commanded  labels  that  can  be  further  modified  as  needed. 
This  is  the  critical  aspect  of  the  concept  that  allows  this  form  of  adaptable  automation  not  to  increase  the 
workload  associated  with  delegation,  much  as  a  sports  team  has  an  approved  set  of  plays  that  facilitate  task 
delegation  by  the  team  leader.  For  example,  the  quarterback  in  football  selects  plays  that  are  executed  by  the 
team  members  (the  other  players). 

Support  for  the  efficacy  of  the  Playbook  approach  to  adaptable  automation  has  come  from  two  sources. 
First,  an  example  Playbook  prototype  for  mission  planning  tool  for  commanding  Unmanned  Combat  Air 
Vehicles  (UCAVs)  has  been  developed  as  a  proof-of-concept  [37,38].  Second,  initial  experimental  studies  of 
the  effects  of  Playbook  interfaces  on  human  performance  have  been  carried  out  [39,40].  These  studies 
examined  the  use  of  a  simple  Playbook  interface  on  system  performance  during  simulated  human-robot 
teaming  using  the  RoboFlag  simulation  environment.  The  RoboFlag  Playbook  provides  the  operator  the 
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ability  to  command  simulated  robots,  individually  or  in  groups,  at  two  levels  of  granularity:  via  providing 
designated  endpoints  for  robot  travel  or  via  commanding  higher  level  behaviors  (or  modes  or  plays)  such  as 
“Patrol  Border.”  The  RoboFlag  simulation  was  modified  to  emulate  a  typical  unmanned  vehicle  (UV)  mission 
involving  a  single  operator  managing  a  team  of  robots.  The  simulated  mission  goal  was  to  send  the  robots 
from  a  home  area  into  enemy  territory,  access  and  obtain  a  specified  target,  and  return  home  as  quickly  as 
possible  with  minimum  loss  of  assets.  In  the  Parasuraman  et  al.  [39]  study,  the  Playbook  interface  was 
evaluated  as  a  function  of  two  sources  of  task  demand  adversary  “posture”  in  which  the  enemy  engagement 
style  was  changed  unpredictably  between  three  types,  offensive,  defensive,  or  mixed;  and  environmental 
uncertainty,  as  manipulated  by  variation  in  the  effective  visual  range  of  each  robotic  vehicle  under  the  control 
of  the  operator.  The  results  showed  that  the  multi-level  tasking  of  the  simplified  Playbook  interface  allowed 
effective  user  supervision  of  robots,  as  evidenced  by  the  number  of  missions  successfully  completed  (percent 
games  won)  and  the  time  for  mission  execution.  As  expected,  significantly  fewer  games  were  won  when  the 
opponent  posture  was  mixed  rather  than  entirely  offensive.  Nevertheless,  users  still  won  a  moderately  high 
proportion  of  games  (about  62%)  and  in  a  relatively  short  time  (about  51  seconds)  in  the  mixed  posture 
condition.  These  findings  suggest,  but  do  not  prove,  that  the  Playbook  interface,  as  a  simple  example  of  a 
delegation  interface,  allowed  users  to  respond  effectively  to  unexpected  changes  in  opponent  posture  by 
tasking  robots  appropriately.  In  a  subsequent  study  [40],  the  Playbook  interface  was  pitted  against  less  flexible 
interfaces  and  to  manual  control,  and  was  found  to  have  significant  benefits  over  both.  Further  confirmation  of 
the  efficacy  of  the  Playbook  approach  to  adaptable  automation  requires  studies  in  which  more  complex 
versions  of  the  Playbook  interface  are  evaluated. 


7.4.3.3  Human  Interaction  with  Adaptive  Systems 

Since  the  theoretical  frameworks  for  adaptive  automation  proposed  by  Rouse  [41]  and  Parasuraman  et  al.  [23], 
there  has  a  steady  stream  of  empirical  work  aimed  at  examining  the  effects  of  adaptive  automation  on  human 
and  system  performance  in  different  application  domains.  The  initial  studies  were  designed  to  investigate 
whether  the  performance  costs  of  certain  forms  of  static  automation  (described  previously),  such  as  reduced 
situation  awareness,  complacency,  skill  degradation,  etc.,  can  be  mitigated  by  adaptive  automation.  Most  of 
these  studies  used  either  a  critical  event  or  model-based  approach  to  adaptive  automation.  A  task  was 
allocated  dynamically  to  either  human  or  machine  control  at  some  point  in  time  during  a  simulated  mission, 
either  when  some  critical  event  occurred,  or  as  dictated  by  a  simple  model  of  operator  and  system 
performance.  For  example,  Hilburn  et  al.  [42]  examined  the  effects  of  adaptive  automation  on  the 
performance  of  military  air  traffic  controllers  who  were  provided  with  a  decision  aid  for  determining  optimal 
descent  trajectories  of  aircraft  -  a  Descent  Advisor  (DA).  The  DA  was  either  present  at  all  times 
(static  automation)  or  came  on  only  when  the  traffic  density  exceeded  a  threshold.  Hilburn  et  al.  found 
significant  benefits  for  controller  workload  (as  assessed  using  pupillometric  and  heart  rate  variability 
measures)  when  the  DA  was  provided  adaptively  during  high  traffic  loads,  compared  to  when  it  was  available 
throughout  (static  automation)  or  only  at  low  traffic  loads.  In  addition  to  physiological  measures  of  workload, 
other  measures  can  also  be  used  to  assess  the  workload-leveling  effect  of  adaptive  automation.  Kaber  and 
Riley  [43],  for  example,  used  a  secondary-task  measurement  technique  to  assess  operator  workload  in  a  target 
acquisition  task.  They  found  that  adaptive  computer  aiding  based  on  the  secondary-task  measure  enhanced 
performance  on  the  primary  task. 

The  results  of  these  and  other  studies  (see  [44]  for  a  review)  indicate  that  adaptive  automation  can  serve  to 
reduce  the  problem  of  unbalanced  workload,  with  attendant  high  peaks  and  troughs,  that  static  automation 
often  induces.  As  discussed  previously,  under  high  workload  operators  tend  to  adopt  an  attention  allocation 
strategy  that  results  in  diminished  monitoring  of  an  automated  task  [11,45].  As  a  result,  operators  can  miss 
malfunctions  in  the  task,  or  fail  to  correct  suboptimal  performance  by  the  automation  because  they  are  busy 
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attending  to  other  tasks.  Adaptive  automation  in  the  form  of  a  temporary  return  of  the  automated  task  to 
human  control  can  mitigate  this  so-called  complacency  effect.  In  a  study  with  the  Multi  Attribute  Test  (MAT) 
flight  simulation  battery,  Parasuraman  et  al.  [46],  showed  that  temporary  return  of  an  automated  engine- 
systems  task  to  human  control  benefited  subsequent  operator  monitoring  of  the  task  when  it  was  returned  to 
automated  control.  It  is  important  to  emphasize  that  the  reallocation  to  human  control  was  brief.  If  the  benefit 
could  only  be  obtained  by  prolonged  human  intervention  in  the  task,  that  would  defeat  the  purpose  of 
automating  the  task  in  the  first  place.  Parasuraman  et  al.  found  that  the  benefit  of  adaptive  reallocation  was 
found  for  either  of  two  methods  of  invocation,  a  model-based  approach  in  which  the  temporary  return  to 
human  control  was  initiated  at  a  particular  time  specified  by  the  model;  and  a  performance-measurement 
approach  in  which  the  adaptive  change  was  triggered  only  when  the  operator’s  performance  on  the  engine- 
systems  task  fell  below  a  specified  level.  A  subsequent  study  showed  that  the  operator  (and  system) 
performance  benefit  could  also  be  sustained  for  long  periods  of  time,  in  principle  indefinitely,  by  repetitive  or 
multiple  adaptive  task  allocation  at  periodic  intervals  [47].  Such  brief,  periodic,  adaptive  reallocation  of  an 
automated  task  to  human  control  can  enhance  overall  system  performance  by  either  maintaining  the  operator’s 
awareness  of  the  automated  task  parameters  or  by  refreshing  the  operator’s  memory  (his  or  her  “mental 
model”)  of  the  automated  task  behavior.  In  support  of  the  latter  explanation,  Farrell  and  Lewandowsky  [48] 
showed  that  they  could  successfully  computationally  model  the  complacency  effect  and  the  benefit  of 
adaptive  reallocation  in  a  three-layer  connectionist  network  with  a  memory  decay  function  for  nodes 
representing  automation  performance. 

These  results  show  that  adaptive  automation  can  balance  operator  workload  and  reduce  automation 
complacency.  However,  Parasuraman  et  al.  [46]  also  showed  that  performance  benefits  can  be  eliminated  if 
adaptive  automation  is  implemented  in  a  clumsy  manner,  supporting  the  concerns  of  Billings  and  Woods  [31]. 
Moreover,  these  studies,  while  clearly  pointing  to  the  potential  benefit  of  adaptive  automation,  had  some 
limitations.  The  model-based  invocation  method  used  in  many  of  the  studies  has  the  advantage  that  the  model 
can  be  implemented  off-line  and  easily  incorporated  into  rule-based  expert  systems.  However,  this  method 
requires  a  valid  model,  and  many  models  may  be  required  to  deal  with  all  aspects  of  human  operator 
performance  in  complex  task  environments. 

7.4.4  Physiological  Measurement  Techniques 

There  are  many  ways  in  which  the  system  changes  can  be  initiated:  subjective  workload  assessments,  operator 
performance  and  physiological  measures.  The  final  section  will  discuss  physiological  measurement  processes 
and  their  relationship  to  adaptation.  Operator  physiological  assessment  offers  another  potential  input  for 
adaptive  systems  [49,23].  Physiological  measures  can  provide  additional  information  that  can  be  tapped  for 
control  of  adaptive  systems.  Technology  is  available  to  measure  a  number  of  physiological  signals  from  the 
operator,  from  autonomic  measures  such  as  heart  rate  variability  to  central  nervous  system  measures  such  as 
the  EEG  and  event-related  potentials  or  ERPs,  as  well  as  measures  such  as  eye  scanning  and  fixations. 
Measurement  technology  is  developing  rapidly  showing  improvements  in  the  areas  of  non-intrusiveness, 
precision  and  prediction.  For  these  reasons,  the  authors  felt  it  was  important  to  survey  these  methods  in  some 
detail.  The  goal  of  using  physiological  methods  for  adaptive  automation  is  to  enhance  operator  performance 
by  restructuring  the  environment  and  or  adjusting  the  demands  of  the  situations  [50].  The  emphasis  is  on  the 
operator’s  capabilities  not  the  system’s  [49]. 

One  way  in  which  systems  may  be  able  to  enhance  operator  performance  is  via  online  measures  of  workload. 
According  to  Scerbo,  Freeman,  and  colleagues  [51]  mental  workload  is  the  critical  factor  in  determining  when 
and  what  type  of  system  changes  need  to  be  made.  There  are  individual  differences  in  how  a  person  handles 
the  demands  of  a  situation  and  these  individual  differences  may  impact  on  performance.  According  to  Gopher 
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and  Donchin  [52]  people  often  increase  their  mental  and  physical  effort  as  task  load  increases.  The  concept  of 
workload  is  used  to  account  for  the  aspect  of  the  interaction  between  the  person  and  the  task  that  cause  task 
demands  to  exceed  the  person’s  capacity  to  successfully  complete  the  task.  The  concept  of  workload  implies 
that  there  are  limitations  in  information  processing  capacity.  Two  theories  form  the  foundation  of  much  of  the 
research  conducted  on  adaptive  automation,  Kahneman’s  Capacity  and  Resource  Theory  and  Wicken’s 
Multiple  Resource  Theory  [22].  According  to  Kahneman,  information  processing  resources  are  limited, 
but  are  a  function  of  arousal.  As  task  load  increases,  arousal  increases  beyond  an  optimal  level  causing 
cognitive  capacity  to  decrease.  In  single  resource  theory,  capacity  can  be  allocated  to  different  tasks  but  the 
result  is  a  depletion  of  overall  capacity.  According  to  Wicken’s  multiple  resource  theory,  information 
processing  resources  are  limited  but  this  limitation  is  a  function  of  the  type  of  mental  resource  being  used 
(i.e.,  visual,  auditory).  Capacity  decreases  more  when  two  task  are  sharing  the  same  resources  than  when  they 
are  using  different  resources.  The  models  are  not  mutually  exclusive;  Wickens,  Dixon  and  Chang  [27] 
recently  found  that  a  combination  of  single  resource  and  multiple  resource  models  best  described  operator 
performance  for  controlling  two  unmanned  vehicles  in  a  multi-task  environment. 

There  is  now  a  substantial  literature  indicating  that  different  psychophysiological  measures  can  be  used  for 
real-time  assessment  of  mental  workload  [53-56].  Researchers  have  examined  the  use  of  Heart  Rate  (HR), 
Heart  Rate  Variability  (HRV),  Electroencephalography  (EEG),  Event  Related  Potentials  (ERP),  Transcranial 
Doppler  Sonography  (TCD),  Functional  Magnetic  Resonance  Imaging  (fMRI)  and  more  recently  Functional 
Near  Infrared  Imaging  (fNIR).  Prinzel,  Freeman,  Scerbo,  Mikulka,  and  Pope  [57]  have  also  specifically 
demonstrated  the  feasibility  of  an  adaptive  system  based  on  EEG  measures.  Each  of  these  measures  has 
advantages  and  disadvantages  and  we  discuss  their  potential  utility  in  an  adaptive  automation  system. 
An  overview  of  the  most  likely  indices,  limitations  and  advantages  is  presented  in  Table  7-4.  A  more  detailed 
review  can  be  found  in  Scerbo,  Freeman,  Mikulka,  Parasuraman,  DiNocera  and  Prinzel  [51]. 
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Table  7-4:  Matrix  of  Physiological  Measures  -  Advantages  and  Disadvantages 


Measures 

What  does  it  measure? 

Advantages 

Disadvantages 

How  Can  The  Measure 
Be  Used? 

Electroencephalography 

EEG 

-Electrical  activity  of 
neuronal  assemblies 
-  Recorded  potential 
reflects  cortical  activity  in 
area  under  electrode 

-Good  temporal  resolution 
-Localization  of  a  behavior  to  a 
cortical  region 
-Extensive  research  base 
-New  systems  more  field 
practical 

-Affected  by  artifacts 
such  as  muscle  activity 
and  heart  beats 
-Poor  spatial  resolution 

-Assess  cortical 
involvement  in  different 
contexts  and  situations 
(e.g.  effect  of  practice  on 
cerebral  activity) 

Event  Related  Potentials 
(Derived  from  EEG) 

ERP 

-Component  of  EEG  related 
specifically  to  a  stimulus 
-Discrete  time  locked 
responses  to  a  stimulus 

-Good  temporal  resolution 
-Can  examine  time -based 
changes  in  cortical  activity  to  a 
stimulus 

-Affected  by  artifacts 
such  as  eye  blinks  and 
heart  activity 
-Poor  spatial  resolution 

-Assess  time  locked 
changes  in  cortical 
involvement  in  different 
contexts  and  situations 

Transcranial  Doppler 
Sonography 

TCD 

-Blood  flow  velocity  in  the 
main  stem  intra-cranial 
arteries 

-Good  spatial  &  temporal 

resolution 

-Non-invasive 

-Less  restrictive  than  other  brain 
imaging  techniques  (e.g.  EEG) 
-Low-cost  &  field  practical 

-Can  not  identify  the 
specific  brain  area 
utilizing  the  metabolic 
resources  (i.e.  blood  flow) 
-Relatively  new 
measurement  technique 

-Potential  metric  for 
cognitive  readiness 
-Assess  how  information¬ 
processing  resources  are 
utilized  and  distributed  in 
different  situations 

Functional  Magnetic 
Resonance  Imaging 

fMRI 

-BOLD  response  (blood 
oxygenation  level 
response),  i.e.,  cerebral 
blood  flow 

-Excellent  spatial  resolution 
-Extensive  medical  research 
base,  growing  cognitive  science 
base 

-Less  good  temporal 

resolution 

-Expensive 

-Restrictive  environment 

-As  with  EEG,  ERP,  and 
others,  good  for  cognitive 
modeling  validation 

Functional  Near  Infrared 
Imaging  Devices 

fNIR 

-Light  is  transmitted  and  the 
reflected  light  from  the 
cortical  level  is  encoded 
and  reconstructed  into  a 
map  of  brain  activity 

-Non-invasive 

-Less  restrictive  than  other  brain 
imaging  techniques  (e.g.  EEG) 
-Field  practical 

-New  technology 
-Still  in  development 
-Unable  to  measure 
signals  from  deep  brain 
tissue 

-Real  time  assessment  of 
warfighter’s  cognitive 
state  that  can  be  utilized 
in  operational 
environments 

Heart  Rate  Variability 
(Derived  from  an 
electrocardiogram) 

HRV 

-Cardiac  activity 

-Assess  the  variability  in  R-R 
intervals 

-Discriminate  between 
parasympathetic  and  sympathetic 
influences  on  the  heart 

-Influenced  by  respiratory 
changes 

-HRV  related  to  changes 
in  cortical  brain  areas 
-Changes  in  HRV  may  be 
related  to  cognitive  events 
and  individual  differences 
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7.4.4.1  Heart  Rate  and  Heart  Rate  Variability 

Cardiac  activity  is  measured  by  electrocardiogram  (ECG/EKG).  An  ECG  records  the  electric  activity 
generated  by  the  action  potentials  of  the  cardiac  muscle  cells.  Two  measures  of  cardiac  activity  that  can 
derived  from  an  ECG  are  heart  rate  and  heart  rate  variability.  Heart  rate  is  measured  in  beats  per  minute. 
Research  has  shown  that  heart  rate  increases  with  increases  in  workload  demands  [51].  There  is  variability  in 
the  heart  cycle.  This  variability  is  called  Heart  Rate  Variability  (HRV)  which  can  be  measured  in  the  time  or 
frequency  domain.  Research  has  suggested  that  variation  in  HRV  may  be  used  to  differentiate  between  levels 
of  task  difficulty,  the  type  of  task  and  mental  workload.  For  example,  the  more  effort  a  task  requires, 
the  greater  the  suppression  of  HRV.  Nickel  and  Nachreiner  [58]  examined  if  the  .1  Hz  component  of  HRV  can 
discriminate  between  levels  of  workload.  Results  showed  that  the  .1  Hz  component  discriminated  between 
periods  of  work  and  rest.  However,  HRV  was  not  different  between  types  of  tasks  or  level  of  task  difficulty 
suggesting  that  tasks  demands  must  be  very  high  for  the  HRV  components  to  be  able  to  discriminate  between 
workload  conditions.  In  general,  cardiac  activity  is  probably  the  most  commonly  used  measure  in  workload 
assessment  [59].  HR  and  HRV  may  reflect  energetic  arousal,  emotional  processes  and  cognitive  processes  that 
may  impact  on  task  performance  [58].  Cardiac  activity  can  be  easily  measured  and  with  the  development  of 
small  telemetric  systems  it  is  a  field  practical  measure. 

7.4.4.2  Electrocortical  Activity 

7. 4. 4. 2.1  Electroencephalography  (EEG) 

EEG  is  a  non-invasive  recording  of  the  fluctuations  in  electrical  activity  of  large  ensembles  of  neurons  in  the 
brain  which  is  taken  from  the  scalp.  Activity  can  be  measured  from  numerous  locations  and  across  various 
bandwidths  (e.g.,  alpha  8-12  Hz).  There  is  an  extensive  research  base  on  EEG  activity  during  various 
cognitive  tasks.  For  example,  Gevins,  Smith,  Leong,  McEvoy,  Du  and  Rush  [53]  showed  that  neural  networks 
could  be  used  to  discriminate  differences  in  memory  load  based  on  EEG.  In  the  adaptive  automation  literature, 
a  general  assumption  is  made  that  changes  in  EEG  reflect  arousal  and  workload  [51].  Pertinent  to  our  research 
interests,  Scerbo  and  colleagues  (e.g.,  [57])  have  conducted  an  elegant  series  of  experiments  which  use  EEG 
to  drive  adaptive  automation.  It  is  a  closed-loop  system  that  moderates  workload  by  decreasing  the  task 
demands  when  workload  increases.  Increases  in  workload  are  assessed  via  EEG.  EEG  is  measured  and  an 
EEG  engagement  index  is  derived  from  components  of  the  frequency  domain.  The  system  allocates  the  tasks 
based  on  the  engagement  index.  A  high  engagement  index  is  related  to  a  high  state  of  alertness  and  an 
increased  ability  to  attend  to  stimuli  [60].  The  MAT  task  is  used  as  the  test-bed.  There  is  a  monitoring  task, 
resource  management  task,  communications  task  and  a  tracking  task.  The  tracking  task  shifts  between  manual 
and  automated  depending  on  the  engagement  ratio  of  the  operator.  Tracking  performance  improved  using  an 
adaptive  policy  wherein  high  engagement  EEG  ratios  invoked  automation  and  low  ratios  invoked  manual 
tracking  compared  to  the  opposite  invocation  policy  (non-adaptive  -  switch  to  auto-low  and  manual-high). 
Prinzel  et  al.  [57]  showed  that  when  the  engagement  index  increased  and  the  system  automated  the  task, 
performance  on  the  tracking  task  was  better  than  when  the  task  was  always  in  the  manual  condition.  Further, 
Prinzel,  Pope  and  Freeman  [61]  provided  biofeedback  about  the  participants’  performance  and  engagement 
level,  which  improved  performance  on  the  tracking  task  during  automation  and  when  the  task  was  returned  to 
manual.  The  engagement  index  modifies  the  system  to  meet  the  real-time  needs  of  the  operator  and  as  a  result 
improves  performance.  Using  biofeedback,  the  participant  was  aware  of  his/her  state  which  allowed  the 
participant  to  be  an  active  participant  in  the  task  environment. 

However,  these  studies  failed  to  show  that  other  invocation  policies  or  simply  automating  the  tracking  task 
would  not  have  been  as  effective.  For  example,  a  recent  study  by  Mikulka,  Scerbo  and  Freeman  [50]  used  a 
similar  paradigm  to  investigate  vigilance  performance  under  low,  medium  and  high  event  rates.  They  also 
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introduced  a  control  to  ensure  that  the  invocation  policies  based  on  the  EEG’s  were  the  cause  of  the 
performance  differences.  Each  EEG  participant  had  a  yoked  partner  who  switched  to  automation  (or  manual) 
mode  in  the  same  time  hack  and  task  level  (unrelated  to  his  or  her  EEG  state)  as  the  EEG  partner.  The  group 
that  had  an  adaptive  EEG  policy  was  again  superior  to  the  non-adaptive  EEG  engagement  policy  group. 
However,  the  yoked  groups  followed  the  same  pattern  implying  that  the  results  were  caused  by  task  difficulty 
and  switching  policy  and  not  the  EEG  indices  per  se.  This  suggest  that  modeling  the  task  environment  might 
be  as  effective  as  using  the  EEG  engagement  policy.  In  summary,  EEG  based  invocation  policies  show 
promise,  but  more  research  needs  to  be  conducted  to  confirm  their  superiority  in  complex  task  environments 
in  comparison  to  automated  systems  and  to  other  invocation  policies. 

7. 4. 4. 2. 2  Event  Related  Potentials  (ERP) 

An  EEG  based  system  such  as  the  one  described  above  is  triggered  by  gross  changes  in  the  level  of 
engagement  and  may  not  be  sensitive  to  different  types  of  task  and  levels  of  task  difficulty  [63].  An  ERP  is  a 
component  of  EEG  related  specifically  to  the  presentation  of  a  stimulus.  It  is  an  average  of  the  EEG  at  the 
electrode  sites  of  interest;  a  time  locked  response  to  a  stimulus.  ERPs  are  characterized  by  their  polarity 
(positive  or  negative)  and  time  of  occurrence  (latency  from  the  onset  of  the  stimulus).  An  ERP  may  reflect 
various  information  processing  activities  such  as  attention,  intention  and  expectation.  For  example,  the  P300 
reflects  the  availability  and  distribution  of  information  processing  resources  and  it  may  be  sensitive  to  changes 
in  workload.  Prinzel  et  al.  [62]  assessed  the  sensitivity  of  the  P300  to  changes  in  workload  and  performance. 
Participants  completed  the  MAT  task  (with  or  without  adaptive  automation)  and  an  auditory  oddball  task  in 
which  they  counted  the  number  of  high  tones.  Performance  was  better  on  the  tracking  and  auditory  task  in  the 
adaptive  automation  condition  relative  to  the  control  and  yoked  control.  Further,  the  amplitude  of  the  P300 
was  greater  in  the  adaptive  automation  condition  than  the  control  and  yoked  control  condition.  The  amplitude 
of  the  P300  may  be  proportional  to  the  attentional  resources  “invested”  in  the  task.  Kramer,  Trejo  and 
Humphrey  [54]  suggest  that  ERPs  may  be  used  to  detect  variations  in  workload.  Kramer  et  al.  had  participants 
complete  a  radar  monitoring  task  during  which  time  task-irrelevant  auditory  probes  were  presented.  The  ERP 
amplitude  (i.e.,  N100,  N200),  elicited  by  the  deviant  auditory  probe,  decreased  as  task  load  increased.  Results 
showed  that  ERPs  are  sensitive  to  task  type  and  level  of  task  difficulty.  Further,  Prinzel  and  colleagues  [62] 
suggest  that  the  P300  may  be  an  indicator  of  the  efficacy  of  the  adaptive  automation  and  may  also  be  used  as  a 
trigger  for  it. 

In  summary,  electrocortical  activity  has  been  used  with  limited  success  in  adaptive  automation.  It  is  important 
to  note  that  as  with  all  measurement  techniques,  EEG  and  ERPs  are  not  without  limitations.  These 
measurements  are  not  sensitive  enough  to  assess  activity  in  narrow  cortical  regions.  EEG  and  ERPs  have  poor 
spatial  resolution.  Further,  the  EEG  signal  can  affected  by  artifacts  such  as  muscle  activity  and  heartbeats. 
Filtering  techniques  can  be  used  to  eliminate  some  of  the  artifacts  from  the  EEG.  Researchers  must  be 
cautious  when  filtering  their  data  that  they  do  not  filter  out  the  EEG  signal  as  well  as  the  artifact.  Current  EEG 
systems  are  not  field  practical,  but  advances  in  technology  are  being  made  and  systems  are  being  designed  to 
be  more  robust  and  field-ready. 

7.4.4.3  Blood  Flow 

7.4.43.1  Positron  Emission  Tomography  (PET)  and  Functional  Magnetic  Resonance  Imaging  (fMRI) 

Brain  imaging  studies  using  PET  and  fMRI  measures  of  cerebral  blood  flow  have  revolutionized  cognitive 
psychology.  There  is  hardly  a  domain  of  cognition  where  such  measures  have  not  been  used  to  localize 
elements  of  cognitive  function  within  the  brain.  It  is  therefore  not  surprising  that  researchers  have  also 
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attempted  to  examine  whether  these  measures  might  be  used  for  investigating  more  applied  aspects  of 
cognition  such  as  mental  workload. 

The  notion  of  mental  workload  as  reflecting  how  hard  one’s  mind  is  working  at  any  given  moment  is 
intuitively  appealing.  Given  that  the  mind  is  a  function  of  the  brain,  it  follows  that  mental  work  should  be 
associated  with  brain  work  [63].  Brain  work  can  be  linked  to  both  global  and  local  changes  in  cerebral  blood 
flow  associated  with  mental  activity.  Sir  Charles  Sherrington,  the  great  19th  century  physiologist,  first 
suggested  that  brain  work  was  related  to  the  regulation  of  the  blood  supply  of  the  brain.  Sherrington 
demonstrated  that  there  is  a  close  coupling  between  the  electrical  activity  of  neuronal  cells,  the  energy 
demands  of  the  associated  cellular  processes,  and  regional  blood  flow  in  the  brain.  His  pioneering  work 
suggested  (but  did  not  prove,  since  he  lacked  the  technology)  that  if  mental  activity  results  in  increased 
neuronal  response  in  localized  regions  of  the  brain,  then  mental  work  could  be  assessed  by  measuring  regional 
cerebral  metabolism  and  blood  flow.  Autoradiographic  studies  in  animals  later  confirmed  Sherrington’s 
principle  for  the  regulation  of  brain  blood  flow  and  its  coupling  to  neuronal  activity  and  energy  usage  -  but  it 
would  take  several  years  before  sensitive  techniques  were  developed  for  measuring  regional  brain  blood  flow 
in  humans.  The  development  of  PET  paved  the  way  for  less  invasive  measurement  of  regional  cerebral 
metabolism  and  blood  flow  in  humans.  PET  is  an  adaptation  of  autoradiographic  techniques  originally 
developed  for  measuring  blood  flow  in  animals.  Regional  cerebral  glucose  metabolism  can  be  non-invasively 
determined  using  PET  and  radioactively  labeled  glucose  (18-fluoro-deoxyglucose),  while  regional  cerebral 
blood  flow  may  be  assessed  with  PET  and  radioactively-labeled  oxygen  (0-15)  in  water. 

PET  is  more  accurate  than  the  older  methods  in  localizing  the  specific  cortical  regions  activated  by  cognitive 
task  demands.  Nevertheless,  the  spatial  resolution  of  PET,  particularly  in  individual  subjects,  could  be 
improved.  Furthermore,  the  need  for  ionizing  radiation,  although  safe  when  used  within  exposure  limits,  is  an 
impediment  against  frequent  use  in  studies  with  normal  human  subjects.  The  recent  development  of  fMRI  has 
overcome  both  these  limitations.  fMRI  provides  non-invasive,  high-resolution  assessment  of  regional  cerebral 
blood  flow. 

How  do  PET  and  fMRI  compare  with  EEG  or  ERP  measures  of  cognition  and  mental  workload?  Because 
brain  electrical  activity  is  recorded  from  the  scalp,  the  EEG  or  the  ERP,  while  having  excellent  temporal 
resolution  in  identifying  the  neural  correlates  of  mental  processing,  do  not  provide  strong  evidence  for  the 
localization  of  neural  activity  associated  with  mental  workload.  The  poor  spatial  resolution  of  EEG  and  ERPs 
can  be  overcome  to  a  degree  through  the  use  of  such  techniques  as  dipole  modeling  and  spatial  deconvolution. 
For  example  Gevins  et  al.  [64]  reported  that  midline  frontal  EEG  theta  activity  is  a  correlate  of  mental 
workload.  This  EEG  signal  is  thought  to  be  generated  in  the  anteromedial  frontal  cortex,  possibly  the  anterior 
cingulate  cortex  which  has  also  been  proposed  to  be  a  high-level  central  executive  control  center  on  the  basis 
of  PET  and  lesion  studies.  In  general,  however,  brain  imaging  techniques  offer  superior  spatial  resolution  to 
EEG/ERPs  and  have  provided  recent  evidence  on  the  cortical  localization  of  mental  workload. 

7. 4. 4. 3. 2  Blood  Flow  Velocity 

Cerebral  blood  flow  velocity  (BFV)  is  assessed  by  Transcranial  Doppler  Sonography  (TCD).  TCD  is  used  to 
assess  the  hemodynamic  changes  in  the  major  cerebral  arteries  [65].  More  specifically,  TCD  measures 
moment-to-moment  changes  in  blood  flow  velocity  (BFV).  BFV  values  are  obtained  by  recording  frequency 
shifts  in  the  ultrasound  that  is  reflected  from  the  blood  flowing  the  cerebral  artery.  The  frequency  shifts  are 
recorded  by  spectral  analysis. 

TCD  could  possibly  be  used  to  ‘quantify  attentional  effort’  [66].  Changes  in  BFV  may  reflect  the  changes  in 
attention.  According  to  the  resource  utilization  model,  demanding  tasks  consume  more  processing  resources 
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than  less  demanding  tasks  [67].  When  there  is  a  decrease  in  performance  this  may  reflect  a  depletion  of  the 
information  processing  resources.  The  amount  of  resources  available  may  be  related  to  BFV.  For  example, 
Maybelen  [68]  showed  that  as  performance  on  vigilance  tasks  declined,  there  was  a  decrease  in  blood  flow 
velocity  over  time.  The  amount  of  blood  flow  was  also  dependent  on  the  type  of  task  utilized;  an  absolute 
judgment  task  was  associated  with  greater  amount  of  blood  flow  and  a  greater  decrease  in  blood  flow  over 
time  than  a  comparative  judgment  task. 

Hitchcock  et  al.  [66]  compared  the  blood  flow  during  a  vigilance  task.  Participants  were  presented  low  or  high 
salience  cues  that  were  presented  at  various  intervals  during  the  experiment  (e.g.,  high  salience  cues  were 
presented  40%  of  the  time).  Results  showed  that  for  low  salience  cues  there  was  a  greater  amount  of  blood 
flow  and  a  greater  decrease  in  blood  flow  over  time  in  the  right  cerebral  artery  relative  to  that  for  high  salience 
cues  which  were  detected  more  often.  The  authors  suggested  that  blood  flow  may  be  an  index  of  the 
expenditure  of  information  processing  resources,  which  is  greater  for  low  than  high  salience  cues. 

TCD  is  a  low  cost  non-invasive  diagnostic  tool.  Unlike  other  measures  of  brain  activity,  such  as  fMRI  or 
EEG,  TCD  has  good  temporal  resolution  while  placing  participants  under  less  restrictive  conditions. 
TCD  may  be  able  to  be  used  in  a  laboratory  or  field  setting.  The  use  of  TCD  would  allow  researchers  the 
flexibility  to  examine  military  issues  in  an  appropriate  context  without  the  restrictions  imposed  by  other 
measurement  techniques.  The  disadvantage  of  TCD  is  that  it  has  poor  spatial  resolution;  researchers  are  only 
able  to  generalize  the  BFV  to  a  cerebral  artery.  The  cortical  region  that  is  utilizing  the  metabolic  resources  can 
not  be  identified.  Furthermore,  this  is  a  relatively  new  technique  in  cognitive  psychology  and  has  not  been 
used  in  adaptive  automation  applications. 

7.4.4.4  Hybrid  Measures 

It  is  possible  that  one  physiological  measure  will  not  be  able  to  capture  the  complexity  inherent  in  human 
performance.  Wilson,  Lambert  and  Russell  [69]  used  a  multiple  measure  approach  to  design  a  physiologically 
based  adaptive  automotive  system.  Wilson  et  al.  had  participants  complete  the  MAT  task  with  two  levels  of 
difficulty  (varied  the  number  of  events  that  occurred  in  five  minutes).  EEG,  ECG,  EOG  and  respiration  were 
measured  during  the  task.  Wilson  et  al.  trained  an  artificial  neural  network  (ANN)  to  recognize  the 
physiological  patterns  that  differentiate  states  of  rest,  low  task  difficulty  and  high  task  difficulty.  The  ANN 
was  then  used  to  determine  which  condition  a  participant  was  performing  and  when  the  high  difficulty  task 
was  detected  the  monitoring  and  auditory  task  were  automated.  Results  showed  that  the  ANN  correctly 
identified  the  task  conditions  and  when  adaptive  automation  was  implemented  tracking  error  decreased  and 
performance  on  the  resource  management  task  increased  compared  to  the  manual  condition.  No  comparison 
was  made  between  fully  and  adaptively  automated  performances.  (Wilson  et  al.). 


7.4.4.5  Measurement  Conclusions 

Most  of  the  measures  were  based  on  the  premise  that  the  measures  are  physiological  correlates  of  workload 
and  that  as  the  human  operator’s  workload  increases  beyond  a  certain  point  performance  will  start  to 
deteriorate.  Correlations  with  perceived  workload  and  increased  task  difficulty  have  been  shown  for  a  number 
of  the  above  indices.  Some  are  currently  not  usable  in  a  combat  environment  and  others  are  not  precise 
enough  to  be  useful  in  their  current  configuration.  However,  these  limitations  are  being  overcome  rapidly  and 
the  developing  technology  will  soon  be  practical  and  precise  enough  for  future  combat  systems.  One  of  the 
most  promising  technologies  is  one  of  the  oldest:  EEG.  However,  the  initial  results  are  pertinent  to  artificial 
task  environments  and  the  usefulness  of  an  adaptive  systems  based  on  EEGs  has  not  been  established. 
More  research  needs  to  done  in  the  following  areas: 
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1)  Establish  the  usefulness  of  these  measures  for  adaptive  and  adaptable  processes  compared  to  non- 
adaptive  automation  and  manual  control  in  complex  environments. 

2)  Compare  various  adaptive  invocation  policies  in  these  environments. 

3)  Investigate  some  of  the  newer  techniques  (including  the  neural  net  modeling)  as  their  precision  and 
practicality  increases. 

7.4.5  General  Conclusions 

Future  combat  environments  will  be  radically  different  with  robotic  and  automated  systems  becoming  a 
ubiquitous  component  of  future  aerial  and  ground  inventories.  The  authors  discussed  human  performance 
advantages  and  disadvantages  of  battlefield  automation  and  concluded: 

1)  A  review  of  the  human  performance  literature  suggests  that  soldiers  tend  to  both  over  rely  and  under 
rely  on  automated  systems  depending  on  the  following  factors: 

•  Decision  order; 

•  Operator  overload; 

•  False  alarm  rate;  and 

•  Reliability  of  the  system. 

2)  Poorly  designed  automation  resulted  in  loss  of  situation  awareness  and  operator  complacency. 

3)  Adaptive  and  adaptable  automated  systems  were  evaluated  as  flexible  alternatives  to  preset 
automation. 

4)  Various  physiological  measures  were  contrasted  as  potential  components  of  adaptive  systems  in  terms 
of  their  relative  maturity  and  intrusiveness. 

5)  Although  some  of  these  measures  were  extremely  promising  as  non-intrusive  indexes  of  operator 
loading;  more  research  is  needed  before  practical  adaptive  or  adaptable  systems  can  be  fielded. 
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7.5  DESIGNING  FOR  FLEXIBLE  HUMAN-AUTOMATION  INTERACTION: 
PLAYBOOKS  FOR  SUPERVISORY  CONTROL 

As  systems  become  more  complex,  there  is  increasing  temptation  to  control  them  via  “automation”  -  that  is, 
a  device,  machine,  or  system  that  accomplishes,  fully  or  partially,  a  function  which  was  or  could  be  performed 
by  a  human  [1].  Examples  of  this  trend  are  rife  in  domains  ranging  from  aviation  and  air  traffic  management 
to  health  care  and  bioinformatics  [2-4]. 

Automation  can  provide  clear  benefits.  Billings  [2]  documents  payoffs  of  increased  automation  in  commercial 
aviation  in  four  key  areas:  safety,  reliability,  economy,  and  comfort  -  but  automation  has  also  been  shown  to 
pose  novel  problems  for  human  operators  in  some  circumstances:  to  increase  workload  and  training 
requirements,  to  result  in  decreased  situation  awareness  and,  in  specific  instances,  to  cause  accidents 
(e.g.,  [1,5-7]). 

This  ‘double-edged  sword’  of  automation  use  [8]  has  motivated  repeated  questions  about  which  tasks  should 
be  automated  to  which  level  or  degree  for  optimal  control,  performance,  and  safety.  Technologists  tend  to 
push  to  automate  tasks  as  fully  as  possible  -  what  has  been  called  the  ‘technological  imperative’  [9].  Human 
factors  engineers  and  others  concerned  with  safety  and  the  human  role  in  advanced  systems  have  tended  to 
highlight  the  risks  of  increased  automation  (e.g.,  [10-12])  and  to  argue  against  the  use  of  higher  levels  of 
automation,  especially  if  the  human  role  in  the  resulting  system  is  decided  by  default. 

An  approach  to  human-automation  relationships  that  retains  the  benefits  of  automation  while  minimizing  its 
costs  and  hazards  is  needed.  For  reasons  discussed  below,  we  believe  that  such  an  approach  requires  that 
neither  human  nor  automation  be  exclusively  in  charge  of  most  tasks,  but  rather  demands  flexibility  in  the  role 
and  level  of  automation  while  placing  control  of  that  flexibility  in  the  human  operator’s  hands.  This  implies 
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that,  for  most  tasks,  automation  levels  should  be  at  neither  end  of  a  possible  spectrum,  but  rather  at  some 
intermediate,  and  adjustable ,  point.  Human  operators  need  to  be  able  to  delegate  tasks  to  automation, 
and  receive  feedback  on  their  performance,  in  much  the  same  way  that  delegation  can  be  performed  in 
successful  human-human  teams  and  organizations:  at  various  levels  of  detail  and  granularity,  and  with  various 
constraints,  stipulations,  contingencies,  and  alternatives. 

In  this  paper,  we  begin  by  reviewing  the  literature  supporting  the  advantages  of  a  delegation-based, 
intermediate  approach  to  human-automation  relationships.  Then  we  examine  traditional  approaches  to 
characterizing  automation  levels  and  discuss  why  a  delegation  approach  demands  they  be  extended  to 
explicitly  represent  task  decomposition.  Finally,  we  present  one  specific  implementation  of  a  delegation 
approach,  based  on  a  sports  team  ‘playbook’  metaphor. 

7.5.1  Intermediate  Automation  Levels-Costs  of  Automation  Extremes 

The  promised,  and  frequently  realized,  benefits  of  automation  have  generally  been  sufficient  to  argue  against 
relegating  many  tasks  to  the  lowest  levels  of  automation — purely  manual  performance.  The  economic  benefits 
of  automation  are  a  strong,  but  not  the  only,  motivator.  Automation  does  offer  substantial  benefits  in  human 
workload  reduction,  increased  performance  and  safety,  when  it  is  properly  designed  and  used  [1,2,5,7,13,14]. 
The  case  against  applying  automation  wherever,  whenever  and  at  the  highest  levels  technologically  feasible 
has  been  harder  to  make.  Much  research  has  been  devoted  to  showing  the  disadvantages  of  reduced  human 
engagement  in  system  control  and  problem  solving  characteristic  of  high-level  automation  [1,2,7,8,15-22]. 
These  problems  will  be  characterized  next: 

1)  Reduced  Situation  and  System  Awareness:  High  levels  of  automation,  particularly  of  decision-making 
functions,  may  reduce  the  operator’s  awareness  of  system  and  environmental  dynamics  [23,24]. 
Humans  tend  to  be  less  aware  of  changes  when  those  changes  are  under  the  control  of  another  agent 
(whether  automation  or  human)  than  when  they  make  the  changes  themselves  [25].  Mode  errors  also 
illustrate  the  impact  of  automation  on  the  user’s  awareness  of  system  characteristics  [26,27].  Mode 
errors  arise  when  the  operator  executes  a  function  that  is  appropriate  for  a  mode  other  than  the  one  the 
system  is  currently  in  [28].  When  a  system  is  capable  of  automatically  changing  its  mode,  as  is  the 
case  with  aircraft  Flight  Management  Systems  (FMS),  mode  errors  become  more  common  precisely 
because  the  operator  is  less  likely  to  be  aware  of  the  current  system  mode  [26].  Several  aviation 
incidents  and  accidents  have  involved  this  type  of  error  [2,29]. 

2)  Trust,  Complacency,  and  Over-Reliance:  Trust  is  an  indicator  of  how  accurately  the  operator 
understands  the  system  [30,31].  Operators  may  not  use  well-designed,  reliable  automation  if  they 
believe  it  to  be  untrustworthy,  or  they  may  continue  to  rely  on  automation  even  when  it  malfunctions 
if  they  are  overconfident  in  it.  Highly  automated  systems  are  prey  to  both  sorts  of  errors  [7]. 

The  problem  of  excessive  trust  or  complacency  has  been  documented  in  several  studies  showing  that 
students  in  laboratory  experiments  [32],  trained  pilots  in  simulation  [33],  and  experienced  air  traffic 
controllers  using  decision  aiding  automation  [34]  are  poor  at  monitoring  automation  for  occasional 
malfunctions  if  their  attention  is  occupied  with  other  tasks.  In  these  studies,  users  grant  more 
autonomy  to  a  system  than  it  was  designed  to  support  by  almost  always  accepting  recommendations 
even  though  they  are  sometimes  incorrect.  Over-reliance  on  automation  can  also  manifest  as  a 
decision  bias  stemming  from  a  heuristic  to  reduce  the  cognitive  effort  involved  in  solving  a  problem. 
This  heuristic  may  result  in  “automation  bias”  [35]  -  a  tendency  to  uncritically  accept  automation 
recommendations.  Although  reliance  on  automation  may  be  an  effective  strategy  in  many  cases,  over¬ 
reliance  can  lead  to  errors  when  automation  is  less  than  perfect. 
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3)  Skill  Degradation:  If  higher  levels  of  automation  can  result  in  complacency  and  loss  of  situation 
awareness,  it  is  perhaps  not  surprising  that  they  can  also  result  in  skill  degradation  if  allowed  to 
persist.  The  pilots  of  increasingly  automated  aircraft  feared  this  effect  with  regards  to  psychomotor 
skills  such  as  aircraft  attitude  control  [2],  but  it  has  also  been  demonstrated  for  decision  making  skills 
[24].  In  both  cases,  the  use  of  an  intermediate,  lower  level  of  automation  alleviated  skill  degradation  if 
the  skills  had  been  learned  in  the  first  place. 

4)  Unbalanced  Mental  Workload:  Automation  can  sometimes  produce  extremes  of  workload,  either  too 
low  or  too  high.  That  high  levels  of  automation  could  leave  an  operator  bored  and  inattentive  is, 
perhaps,  to  be  expected.  That  automation  can  increase  workload,  on  the  other  hand,  is  one  of  the 
“ironies  of  automation”  [8]  because  many  automated  systems  are  introduced  as  workload-saving 
moves,  but  this  does  not  always  occur.  First,  if  automation  is  implemented  in  a  “clumsy”  manner, 
e.g.,  if  executing  an  automated  function  requires  extensive  data  entry  or  “re -programming”  by  human 
operators  at  times  when  they  are  very  busy,  workload  reduction  may  not  occur  where  it  is  most 
needed  [36].  Second,  if  engagement  of  automation  requires  considerable  “cognitive  overhead”  [37], 
i.e.,  extensive  cognitive  evaluation  of  the  benefit  of  automation  versus  the  cost  of  performing  the  task 
manually,  then  users  may  experience  greater  workload  in  using  the  automation. 

5)  Degraded  Overall  Performance:  The  most  significant  effect  of  using  too  high  an  automation  level 
may  be  degraded  overall  performance  of  the  human  +  machine  system.  In  an  experiment  involving 
aircraft  navigation  and  route  planning,  Layton  et  al.  [38]  provided  operators  with  one  of  three  levels  of 
support  ranging  from  ‘sketching  only’,  where  the  human  sketched  a  desired  route  and  the  system 
provided  feedback  about  its  feasibility,  to  ‘full  automation’  where  the  system  automatically  provided  a 
recommended  ‘best’  route  according  to  its  optimization  criteria.  An  intermediate  level  allowed  the 
user  to  ask  for  a  route  with  specific  characteristics  that  the  system  then  provided  if  possible.  They 
found  that  humans  in  the  intermediate  and  high  automation  conditions  frequently  explored  more 
routes  because,  in  the  highly  manual  sketching  condition,  the  process  of  arriving  at  a  route  was  too 
difficult  to  allow  trying  many  alternatives  consistently  and  fully.  By  contrast,  as  in  [35],  in  the  full 
automation  condition,  users  tended  to  accept  the  first  route  suggested  without  exploring  it  or  its 
alternatives  deeply.  Even  when  they  did  explore,  the  system’s  recommendation  tended  to  narrow  and 
bias  their  search.  Users  tended  to  check  the  route  provided  for  obvious  mistakes  rather  than  generating 
a  preferred  route  on  their  own.  Particularly  in  trials  when  the  automation  performed  suboptimally 
(e.g.,  because  it  failed  to  adequately  consider  uncertainty  in  weather  predictions),  the  intermediate 
level  of  automation  produced  better  overall  solutions. 

6)  Lower  User  Acceptance:  Finally,  an  additional  problem  with  high  levels  of  automation  is  lack  of  user 
acceptance.  This  may  be  particularly  true  of  highly  trained  and  skilled  operators  of  complex,  high- 
criticality  systems  such  as  aircraft,  military  systems,  process  control,  etc.  For  example,  Miller  [39], 
interviewed  several  pilots  and  designers  to  develop  a  consensus  list  of  prioritized  goals  for  a  “good” 
cockpit  configuration  manager  for  the  Rotorcraft  Pilot’s  Associate  [40].  Even  though  this  system 
provided  advanced  automation  capable  of  managing  information  displays  and  cockpit  functions  to 
conform  to  pilot  intent,  two  of  the  top  three  items  on  the  pilots’  consensus  list  were  “Pilot  remains  in 
charge  of  task  allocation”  and  “Pilot  remains  in  charge  of  information  presented.” 

Similarly,  Vicente  [41]  and  Bisantz  et  al.  [42]  cite  examples  of  human  interactions  with  even  such  low 
criticality  automation  as  food  planning  aids  in  fast  food  restaurants,  showing  that  operators  can 
become  frustrated  when  forced  to  interact  with  automation  that  removes  their  authority  to  do  their 
jobs  in  the  best  way  they  see  fit.  Vicente  [41]  summarizes  findings  from  [43]  showing  that  jobs  in 
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which  human  operators  have  high  psychological  demands  coupled  with  low  decision  latitude 
(the  ability  to  improvise  and  exploit  one’s  skills  for  job  performance)  lead  to  higher  incidences  of 
heart  disease,  depression,  pill  consumption,  and  exhaustion. 

7.5.2  Tradeoff  Space  for  Effects  of  Automation  Level 

The  findings  summarized  above  show  that  a  mixture  of  human  and  automation  involvement  is  frequently 
desirable  rather  than  the  extremes  of  full  or  no  automation.  In  these  cases,  human  +  machine  systems  must  be 
designed  for  an  appropriate  relationship  between  operators  and  automation  allowing  both  parties  to  share 
responsibility,  authority  and  autonomy  over  many  work  behaviors  in  a  safe,  efficient  and  reliable  fashion. 
The  effects  of  different  human-automation  relationships  over  a  task  or  function  can  be  viewed  as  a  Tradeoff 
space’.  Figure  7-8  presents  a  conceptual  view  of  this  space  over  three  relevant  parameters: 


w 


Increased  human 
management  (“adaptable”) 


Increased  automation 
management  (“adaptive”) 


Figure  7-8:  The  Tradeoff  Relationship  between  System 
Competency,  Human  Workload  and  Unpredictability. 


The  Competency  of  the  Human-Machine  System.  Competency  refers  to  correct  behavior  in  context. 
Therefore,  a  system  becomes  more  ‘competent’  whenever  it  provides  correct  behaviors  more  frequently  or 
encompasses  a  greater  number  of  contexts. 

The  Mental  Workload  Required  for  the  Human  to  Interact  with  the  System.  Workload  refers  to  the 
combined  amount  of  attentional  and  cognitive  “energy”  the  human  must  exert  to  use  the  system  [44-46]. 
Mental  workload  for  the  human  is  modeled  because  it  is  the  major  constraint  on  the  other  two  dimensions. 

The  Unpredictability  of  the  System  to  the  Human  Operator.  Unpredictability  refers  to  the  inability  of  the 
human  to  know  exactly  what  the  automation  will  do  when.  Unpredictability  is  a  consequence  of  the  human’s 
not  personally  taking  all  actions  in  the  system — of  not  being  ‘in  control’  directly  and  immediately.  It  is 
inversely  correlated  with  situation  awareness  (at  least  awareness  pertinent  to  automation  functions)  and, 
generally,  workload.  Good  system  and  interface  design  and  good  hiring  and  training  practices  can  serve  to 
reduce  unpredictability,  but  any  form  of  task  delegation — whether  to  automation  or  other  humans — must 
result  in  a  degree  of  unpredictability  if  it  offloads  tasks. 

The  triangle  in  Figure  7-8  illustrates  the  relationship,  or  ‘tradeoff  space’,  among  these  three  dimensions. 
The  performance  of  a  set  of  functions  to  a  given  level  of  competency  can  only  be  achieved  through  some  mix 
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of  workload  and  unpredictability.  A  user’s  workload  can  be  reduced  by  allocating  some  functions  to 
automation,  but  only  at  the  expense  of  increased  unpredictability;  conversely,  reducing  unpredictability  by 
having  the  user  perform  functions  increases  workload.  Alternate  designs  for  a  level  of  competency  represent 
different  mixes  of  workload  and  unpredictability  -  corresponding  to  varying  the  lengths  of  the  triangle’s  sides 
while  holding  the  base  constant.  It  is  sometimes  possible  to  reduce  both  workload  and  unpredictability  for  a 
given  level  of  competency  through  better  design  -  corresponding  to  shortening  the  height  of  the  triangle. 

In  this  tradeoff  model,  any  increase  in  human-machine  system  competency  must  affect  the  human  in  that 
either: 

1)  The  added  functionality  must  be  fully  controlled  by  the  human(s),  resulting  in  workload  increases;  or 

2)  It  must  be  managed  by  automation,  resulting  in  unpredictability  increases. 

Opperman  [47]  identified  these  alternatives  as  ‘adaptive’  and  ‘adaptable’  approaches  to  system  design 
(see  also  [48]).  In  either  case,  the  human  +  machine  system  can  adapt  to  various  contexts,  but  in  adaptive 
systems  automation  determines  and  executes  the  necessary  adaptations,  while  in  adaptable  systems,  the 
operator  is  in  charge  of  the  desired  adaptations.  The  persistent  debate  (e.g.,  [49,50])  in  the  Human-Computer 
Interface  community  over  intelligent  agents  vs.  direct  manipulation  interfaces  is  a  similar  manifestation  of 
these  alternative  approaches. 

Another  implication  of  the  tradeoff  space  is  that  these  approaches  are  the  endpoints  of  a  spectrum  with  many 
alternatives  in  between,  each  representing  a  different  tradeoff  between  workload,  unpredictability  and 
competency  (as  in  Figure  7-9).  The  range  of  alternatives  available  may  be  constrained  by  automation  and/or 
human  capabilities,  but  within  the  range  of  feasible  systems,  an  alternative  must  be  selected  to  assign  roles 
and  responsibilities  between  human(s)  and  automation.  This  decision  is  the  process  of  selecting  an  automation 
‘level’  or  relationship  and  it  corresponds  to  picking  a  point  in  the  tradeoff  space  [1].  When  the  division  of 
labor  is  done  by  a  supervisor  for  a  human  team,  the  process  may  be  called  ‘delegation’;  when  done  by  a 
designer  prior  to  system  operation,  it  is  part  of  system  design.  Our  objective  is  to  provide  a  human  supervisor 
the  ability  to  flexibly  delegate  tasks  to  automation  at  the  time  of  use  rather  than  relying  on  a  static  design 
point. 


Flexibility  across  Spectrum 


Figure  7-9:  A  Tasking  Interface  Provides  Flexibility  Within  the  Tradeoff  Space. 
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7.5.2.1  Flexible  Automation  Levels 

Traditionally,  the  chief  difference  between  task  delegation  performed  by  a  supervisor  and  design  of  an 
automated  system  has  been  that  the  supervisor  had  much  more  flexibility  in  what,  when  and  how  to  delegate 
and  better  awareness  of  the  task  performance  conditions,  while  the  designer  had  to  fix  a  relationship  at  design 
time  and  incorporate  it  statically  into  the  to-be-built  system  for  all  contexts  of  use.  More  recently,  “adaptive 
automation”  [48,51-56],  has  been  developed  which  chooses  a  level  of  automation  for  tasks  based  on 
contextual  criteria  such  as  location,  situation,  workload,  experience,  physiological  state,  etc.  In  adaptive 
systems,  the  division  of  labor  between  human  and  machine  is  not  fixed.  An  adaptive  aiding  system  may 
provide  assistance  through  context-dependent  forms  of  automation  including  the  adaptive  presentation  of 
information. 

The  adaptive  automation  concept  was  proposed  over  25  years  ago  [57],  however  the  technologies  for  its 
effective  implementation  have  only  recently  matured  and  empirical  evidence  of  its  effectiveness  been 
provided.  Studies  have  shown  that  adaptive  systems  can  regulate  operator  workload  and  enhance  performance, 
while  preserving  the  benefits  of  automation  [51,58,59].  The  performance  costs  of  certain  forms  of  automation 
described  previously  -  reduced  situation  awareness,  complacency,  skill  degradation,  etc  -  may  also  be 
mitigated  [54,55,60-63].  Adaptive  aiding  systems  have  recently  been  successfully  flight  tested  in  the 
Rotorcraft  Pilot’s  Associate  [64]. 

Such  systems  are  truly  “adaptive”  in  Opperman’s  [47]  sense  since  they  choose  the  level  of  automation  to 
apply.  By  contrast,  human  task  delegation  within  a  team  is  more  nearly  an  “adaptable”  system,  since  the 
human  supervisor  can  choose  which  tasks  to  give  to  a  subordinate,  how  much  to  dictate  about  how  (or  how 
not)  to  perform  subtasks,  how  much  attention  to  devote  to  monitoring,  approving,  reviewing  and  correcting 
task  performance,  etc. 

The  ability  to  delegate  tasks  to  subordinates  -  and  to  do  so  flexibly,  adapting  both  the  tasks  allocated  and  the 
degree  of  supervision,  instruction  and  monitoring  to  the  context  and  the  capabilities  of  team  members  - 
is  clearly  a  powerful  form  of  organizing  and  performing  work.  It  is  also  deeply  familiar  to  anyone  who  has 
worked  in  a  team  or  a  supervisory  setting.  As  described  below,  we  have  sought  to  extend  such  delegation 
capabilities  to  control  over  flexible  automation.  Before  presenting  that  work,  however,  we  must  first  describe 
what  is  meant  by  an  automation  level  and  then  argue  for  an  extension  to  traditional  definitions  to  support 
delegation-based  approaches  to  supervisory  control. 


7.5.2.2  Characterizing  Automation  Levels 

To  discuss  alternative  forms  of  automation,  it  is  helpful  to  have  a  scheme  for  characterizing  automation  types, 
roles  and  responsibilities.  In  general,  such  characterizations  have  been  made  in  terms  of  ‘levels’  of 
automation:  defining  a  spectrum  of  possible  relationships  ranging  from  full  human  control  authority  to  full 
automation. 

7.5.2.3  Prior  Automation  Level  Spectra 

Bright  [65]  was  perhaps  the  first  to  propose  such  a  spectrum.  Bright  viewed  automation  as  evolving  through 
17  levels  of  competency  and  noted  that  intermediate  stages  might  well  demand  greater  human  workload,  skill 
and  training  requirements  than  lower  levels.  Sheridan  [66],  is  generally  credited  with  defining  “supervisory 
control”  -  a  specific  relationship  where  control  automation  allows  the  human  to  behave  as  if  interacting  with 
an  intelligent,  human  subordinate.  He  described  a  ten  point  spectrum  of  automation  levels  whose  endpoints 
are  full  control  autonomy  for  the  human  (essentially  no  role  for  automation)  and  vice  versa  [66]. 
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The  intermediate  levels  in  this  spectrum,  then,  represent  alternative  forms  of  supervisory  control  interactions. 
A  version  of  this  list  is  shown  in  Table  7-5. 

Table  7-5:  Levels  of  Automation  (after  [66]) 

1  Human  does  it  all 

2  Computer  offers  alternatives 

3  Computer  narrows  alternatives  down  to  a  few 

4  Computer  suggests  a  recommended  alternative 

5  Computer  executes  alternative  if  human  approves 

6  Computer  executes  alternative;  human  can  veto 

7  Computer  executes  alternative  and  informs  human 

8  Computer  executes  selected  alternative  and  informs  human  only  if  asked 

9  Computer  executes  selected  alternative  and  informs  human  only  if  it  decides  to 

10  Computer  acts  entirely  autonomously 


A  problem  with  these  simple,  uni-dimensional  models  of  human-automation  relationships  is  that  they  are 
ambiguous  about  what  the  relationship  is  defined  over.  Parasuraman,  Sheridan,  and  Wickens  [1]  recently 
noted  that  Sheridan’s  levels  referred  mainly  to  automation  which  makes  decisions,  offers  suggestions  and/or 
executes  actions.  There  are,  however,  other  jobs  automation  can  do:  for  example,  sensing  and  processing 
information  to  detect  situations  of  interest.  Parasuraman  et  al.  [1]  applied  a  four-stage  model  of  human 
information  processing  to  arrive  at  four  functions  that  must  be  accomplished  to  perform  most  tasks: 

•  Information  acquisition; 

•  Information  analysis; 

•  Decision  and  action  selection;  and 

•  Action  implementation. 

Since  these  functions  can  be  performed  by  either  human  or  automation  in  various  mixes,  Parasuraman  et  al. 
[1]  added  a  second  dimension  to  Sheridan’s  spectrum.  Most  human  +  automation  systems  can  be 
characterized  by  a  mix  of  levels  of  automation  across  these  four  stages,  as  in  Figure  7-10.  One  system  (A) 
might  be  highly  autonomous  in  information  acquisition,  but  comparatively  low  on  the  other  three  functions, 
while  a  second  system  (B)  might  offer  a  high  level  of  automation  across  all  four  sub-functions. 
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Figure  7-10:  Levels  of  Automation  by  Information  Processing  Phase  for  Two  Systems  (from  [1]). 

7.5.2.4  Extending  Automation  Levels 

An  important  implication  of  this  two-dimensional,  levels  and  stages  model  of  automation  [1]  is  that  a  parent  task 
can  be  decomposed  -  at  least  into  four  stages  -  and  that  a  single  automation  level  need  not  be  applied 
homogenously.  However,  in  this  approach  a  parent  task  is  decomposed  into  abstract  subtasks  based  on 
information  processing  stages,  whereas  other  task  decomposition  methods  arguably  provide  more  insight  into 
how  a  task  may  be  performed.  Chief  among  these  are  functional  decompositions  (e.g.,  Operator  Functional 
Modeling  [67],  Goals-Operations-Methods-Scripts  (GOMS)  models  [68]  and  Plan  Goal  Graphs  [69])  that  stress 
the  sub-functions  required  to  achieve  a  parent,  and  sequential  process  models  (e.g.,  PERT  or  CPM  charts 
(e.g.,  [70],  Petri  nets  [71])  that  stress  the  temporal  ordering  and  duration  of  a  function’s  steps. 
Such  decompositions  are  inherently  hierarchical  and  may  proceed  through  any  number  of  levels  to  some 
primitive,  “stopping”  level  [72]  that  may  be  imposed  by  biology,  physics  or,  more  commonly,  the  functional 
purpose  of  the  decomposition. 

Thus,  while  the  two-dimensional  model  of  automation  roles  offered  by  Parasuraman  et  al.  [1]  represents  a 
major  advance  over  earlier  uni-dimensional  models,  it  arguably  does  not  go  far  enough.  The  subdivision  of  a 
parent  task  into  four  information  processing  phases  represents  only  a  single  level  of  decomposition  into  an 
abstract  set  of  task  categories.  In  practice,  tasks  are  accomplished  by  hierarchical  sequences  of  specific 
activities  -  the  parent  task’s  subtasks.  Automation  may  be  applied  differently  to  each  and  every  subtask  that 
comprises  the  parent  task.  Thus,  the  profile  of  automation  levels  sketched  in  Figure  7-10  should  stretch  not 
merely  over  the  four  information  processing  phases  at  one  level  of  decomposition,  but  over  as  many  subtasks 
and  levels  as  we  want  or  need  to  divide  a  parent  task  into.1 


1  In  fact,  the  relationship  between  automation  level  and  task  decomposition  is  more  complex  than  this.  As  is  well  understood 
[1,2,10,26],  automation  does  not  merely  shift  responsibility  for  tasks  but  can  change  their  nature  as  well.  In  a  task  decomposition, 
this  means  that  some  sub-tasks  may  be  eliminated  while  others  may  be  added.  This  implies  that  there  will  generally  be  multiple 
alternate  decompositions  of  a  parent  task  depending  on,  among  other  things,  what  level  of  automation  is  used.  Each  alternative 
constitutes  a  different  combination  of  human  and  automation  subtasks.  The  particular  set  represented  should  be  a  function  of  the 
model’s  purpose:  if  used  for  design,  the  full  set  of  possible  and  reasonable  alternatives  should  be  explored;  if  used  to  represent 
possible  actions  in  an  existing  system,  then  only  the  available  methods  need  be  represented. 
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When  one  identifies  a  level  of  automation  for  a  system  using  Bright’s  or  Sheridan’s  spectra,  one  is  identifying 
something  like  an  average  or  modal  level  over  the  subtasks  the  system  accomplishes.  Similarly,  when  one 
uses  a  model  of  levels  and  stages  of  automation  [1],  one  is  clustering  the  subtasks  by  information  processing 
stage  and  again  averaging.  Assigning  levels  by  stages  offers  more  sensitivity  than  assigning  them  only  to  the 
parent  task,  but  it  is  still  an  abstraction.  In  practice,  one  could  identify  the  specific  subtasks  to  be  performed 
and  represent  an  automation  level  for  each  of  them. 

Why  would  one  want  to?  More  sub-tasks  are  not  necessarily  better  and,  in  fact,  Parasuraman  et  al.  [1] 
explicitly  chose  a  four-stage  model  over  more  elaborate  alternatives  to  simplify  design  considerations. 
Precision  in  representation  may  be  inherently  desirable  for  some  purposes  (such  as  training  and  detailed 
design),  but  our  interest  is  in  supporting  flexible  task  delegation  with  automation.  As  we  saw  above,  for  any 
intermediate  level  of  automation  for  a  task,  there  are  roles  for  both  humans  and  automation  on  its  sub-tasks. 
Yet,  someone  must  coordinate  those  roles.  Insofar  as  the  human  is  required  to  manage,  or  at  least  be  aware  of, 
that  division  of  labor,  s/he  must  understand  the  decomposition  of  the  task  in  question.  Supervisory  control  is  a 
process  of  task  delegation  and  delegation  requires  task  decomposition.  Task  decomposition  seems  to  reflect 
the  way  humans  delegate  responsibilities  to  subordinates,  and  reason  about  task  performance  when  receiving 
feedback  [74].  Delegation  is,  in  short,  a  process  of  assigning  specific  roles  and  responsibilities  for  the  subtasks 
of  a  parent  task  for  which  the  delegating  agent  retains  authority  (and  responsibility).  Furthermore, 
communication  about  intent  to  the  agent’s  subordinates  is  frequently  in  terms  of  specific  goals,  methods 
and/or  constraints  on  how,  when  and  with  what  resources  the  subtasks  should  be  accomplished  [73-75]. 
For  these  reasons,  it  is  essential  that  those  subtasks  be  modeled  explicitly. 

7.5.2.5  Using  Extended  Automation  Levels  in  Design 

Above,  we  argued  that  an  automation  level  is  a  defined  combination  of  roles  and  responsibilities  between 
human  and  automation  for  a  task  to  be  performed,  and  that  a  task  can  generally  be  decomposed  into  subtasks  - 
each  of  which  may  have  its  own  automation  level.  This  realization  opens  the  doors  for  more  explicit 
communication,  either  during  design  or  operations,  about  the  relationship  between  human  and  automation 
concerning  the  tasks  to  be  accomplished. 

We  are  developing  a  method  for  pushing  this  communication  into  the  run-time  environment  -  more  closely 
emulating  delegation  in  human-human  work  relationships  and  allowing  the  operator  to  smoothly  adjust  the 
‘amount’  and  level  of  automation  used  depending  on  such  variables  as  time  available,  workload,  decisions 
criticality,  trust,  etc. 

While  this  does  not  eliminate  the  tradeoff  presented  in  Figure  7-8,  it  mitigates  it  by  allowing  operators  to 
choose  various  points  on  the  spectrum  for  interaction  with  automation  (Figure  7-9).  The  fundamental  tradeoff 
remains,  but  the  operator  is  in  charge  of  choosing  a  tradeoff  point  in  that  space.  This  strategy  follows  both 
Rasmussen’s  [76]  and  Vicente’s  [41]  approach  of  allowing  the  operator  to  ‘finish  the  design’  at  the  time  and 
in  the  context  of  use.  This  allows  more  control  and  authority  over  how  and  when  the  user  interacts  with 
automation. 

Such  a  system  must  avoid  two  problems.  First,  it  must  make  achievable  the  task  of  commanding  automation  to 
behave  as  desired  without  excessive  workload.  Second,  it  must  ensure  safe  and  effective  overall  behavior. 
We  have  created  a  design  metaphor  and  system  architecture  that  addresses  these  two  concerns.  We  call  the 
general  class  of  systems  that  can  take  delegation  instruction  at  various  levels  tasking  interfaces,  because  they 
allow  posing  a  task  to  automation  in  the  different  ways  one  might  ‘task’  a  knowledgeable  subordinate. 
Examples  we  would  label  tasking  interfaces  can  be  found  in  [75,77].  Our  particular  approach  to  enabling, 
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facilitating  and  ensuring  correctness  from  a  tasking  interface,  we  call  a  Playbook  -  because  it  is  based  on  the 
metaphor  of  a  sports  team’s  book  of  approved  plays.  We  have  been  exploring  alternate  methods  to  achieve  the 
flexibility  and  ease  of  human-human  delegation  within  human-machine  systems.  Two  examples  of  our  work 
on  delegation  systems  -  a  “Playbook®”  for  task-  and  constraint-based  delegation  and  “Policy”  interactions  for 
setting  high  level  priorities  for  automation  -  are  discussed  in  the  next  section. 
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7.6  DELEGATION  ARCHITECTURES:  PLAYBOOKS  AND  POLICY  FOR 
KEEPING  OPERATORS  IN  CHARGE 

We  argue  that  as  Unmanned  Military  Vehicles  become  more  intelligent  and  capable,  and  as  we  attempt  to 
control  more  of  them  with  fewer  humans  in  the  loop,  we  need  to  move  toward  a  model  of  delegation  of 
control  rather  than  the  direct  control  that  characterizes  current  practice.  We  identify  and  describe  five 
delegation  methods  which  can  serve  as  building  blocks  from  which  to  compose  complex  and  sensitive 
delegation  systems:  delegation  through  (1)  providing  goals ,  (2)  providing  full  or  partial  plans ,  (3)  providing 
negative  constraints ,  (4)  providing  positive  constraints  or  stipulations ,  and  (5)  providing  priorities  or  value 
statements  in  the  form  of  a  policy.  We  then  describe  two  implemented  delegation  architectures  that  illustrate 
the  use  of  some  of  these  delegation  methods:  a  “playbook”  interface  for  UAV  mission  planning  and  a  “policy” 
interface  for  optimizing  the  use  of  battlefield  communications  resources. 

7.6.1  UMV  Control  as  Human-Automation  Delegation 

While  Unmanned  Military  Vehicles  (UMVs)  hold  the  promise  of  radical  change  and  improvement  for  a  wide 
range  of  military  applications  they  also  pose  a  host  of  challenging  problems.  Chief  among  these  is  how  to 
enable  a  human  operator,  who  may  well  be  heavily  engaged  in  tasks  of  his  or  her  own,  to  retain  sufficient 
control  over  the  UMV(s)  to  ensure  safe,  efficient  and  productive  outcomes.  This  problem  is,  of  course, 
magnified  when  the  UMVs  may  be  responsible  for  the  lives  of  many  soldiers  or  civilians,  may  be  capable  of 
unleashing  lethal  force  on  its  own,  and  when  a  single  human  may  be  striving  to  control  groups  or  even  swarms 
of  potentially  autonomous  and  independent  actors  and  may  be  concurrently  engaged  in  other,  high  tempo  and 
criticality  tasks  of  his  or  her  own. 

Yet  this  problem  is  not  completely  novel.  Humans  have  been  striving  to  retain  control  and  produce  efficient 
outcomes  via  the  behavior  of  other  autonomous  agents  for  millennia.  It  just  so  happens  that  those  “agents” 
have  been  other  humans.  Not  surprisingly,  we  have  developed  many  useful  methods  for  accomplishing  these 
goals,  each  customized  to  a  different  domain  or  context  of  use.  When  we  have  some  degree  of  managerial 
authority  over  another  human  actor  and  yet  will  not  be  directly  commanding  performance  of  every  aspect  of  a 
task,  we  call  the  relationship  (and  the  method  of  commanding  task  performance)  delegation.  Delegation 
allows  the  supervisor  to  set  the  agenda  either  broadly  or  specifically,  but  leaves  some  authority  to  the 
subordinate  to  decide  exactly  how  to  achieve  the  commands  supplied  by  the  supervisor.  Thus,  a  delegation 
relationship  between  supervisor  and  subordinate  has  many  requirements: 

1)  The  supervisor  retains  overall  responsibility  for  the  outcome  of  work  undertaken  by  the  supervisor/ 
subordinate  team  and  retains  the  authority  commensurate  with  that  responsibility. 

2)  The  supervisor  has  the  capability  to  interact  very  flexibly  and  at  multiple  levels  with  the  subordinate. 
When  and  if  the  supervisor  wishes  to  provide  detailed  instructions,  s/he  can;  when  s/he  wishes  to 
provide  only  loose  guidelines  and  leave  detailed  decision  making  up  to  the  subordinate,  s/he  can  do 
that  as  well  -  within  the  constraints  of  the  capabilities  of  the  subordinate. 

3)  To  provide  useful  assistance  within  the  work  domain,  the  subordinate  must  have  substantial 
knowledge  about  and  capabilities  within  the  domain.  The  greater  these  are,  the  greater  the  potential 
for  the  supervisor  to  offload  tasks  (including  higher  level  decision  making  tasks)  on  the  subordinate. 

4)  The  supervisor  must  be  aware  of  the  subordinate’s  capabilities  and  limitations  and  must  either  not  task 
the  subordinate  beyond  his/her  abilities  or  must  provide  more  explicit  instructions  and  oversight  when 
there  is  doubt  about  those  abilities. 
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5)  There  must  be  a  “language”  or  representation  available  for  the  supervisor  to  task  and  instruct  the 
subordinate.  This  language  must  (a)  be  easy  to  use,  (b)  be  adaptable  to  a  variety  of  time  and 
situational  contexts,  (c)  afford  discussing  tasks,  goals  and  constraints  (as  well  as  world  and  equipment 
states)  directly  (as  first  order  objects),  and  (d)  most  importantly,  be  shared  by  both  the  supervisor  and 
the  subordinate(s). 

6)  The  act  of  delegation  will  itself  define  a  window  or  space  of  control  authority  within  which  the 
subordinate  may  act.  This  authority  need  not  be  complete  (e.g.,  checking  in  with  the  supervisor  before 
proceeding  with  specific  actions  or  using  some  resources  may  be  required),  but  the  greater  the 
authority,  the  greater  the  workload  reduction  on  the  supervisor. 

Items  4  and  6  together  imply  that  the  space  of  control  authority  delegated  to  automation  is  flexible  -  that  the 
supervisor  can  choose  to  delegate  more  or  less  “space,”  and  more  or  less  authority  within  that  space  (that  is, 
range  of  control  options),  to  automation.  Item  5  implies  that  the  language  available  for  delegation  must  make 
the  task  of  delegating  feasible  and  robust  -  enabling,  for  example,  the  provision  of  detailed  instructions  on 
how  the  supervisor  wants  a  task  to  be  performed  or  a  simple  statement  of  the  desired  goal  outcome. 

7.6.2  Types  of  Delegation 

We  have  developed  a  variety  of  architectures  within  which  to  support  human  delegation  interactions  with 
automation.  Of  particular  interest  as  a  core  enabling  technology  is  the  “language”  or  representation  for 
delegation  described  in  item  #5  above.  As  Klein  points  out  [1],  without  successfully  sharing  an  understanding 
of  the  tasks,  goals  and  objectives  in  a  work  domain,  there  can  be  no  successful  communication  of  intent 
between  actors.  We  believe  there  are  five  kinds  of  delegation  actions  or  delegation  methods  that  should  be 
supported  within  such  a  representation,  as  described  in  the  Figure  7-11  below.  Note  that  each  method  forms  a 
building  block  and  they  can  be  combined  into  more  effective  and  flexible  composite  delegation  interactions. 
Note  also  that  the  subordinate  has  a  specific  responsibility  in  response  to  each  method,  as  articulated  in  the 
table  below: 


Table  7-6:  Supported  Delegation  Actions  and  Methods 


Supervisor’s  Delegation  Action 

Subordinate’s  Responsibility 

1 .  Stipulation  of  a  goal  to  be  achieved  -  where  a 
goal  is  a  desired  (partial)  state  of  the  world. 

Achieve  the  goal(s)  if  possible  (via  any  means 
available),  or  report  if  incapable. 

2.  Stipulation  of  a  plan  to  be  performed  -  where  a 
plan  is  a  series  of  actions,  perhaps  with  sequential 
or  world  state  dependencies. 

Follow  the  plan  if  possible  (regardless  of  out-come) 
or  report  if  incapable. 

3.  Provide  constraints  in  the  form  of  actions  or  states 
to  be  avoided. 

Avoid  those  states  or  actions  if  possible,  report  if 
not. 

4.  Provide  “stipulations”  in  the  form  of  actions  or 
states  (i.e.,  sub-goals)  to  be  achieved. 

Achieve  those  states  or  perform  those  actions  if 
possible,  report  if  not. 

5.  Provide  an  “optimization  function”  or  “policy” 
that  enables  the  subordinate  to  make  informed 
decisions  about  the  desirability  of  various  states 
and  actions. 

Work  to  optimize  value  within  the  “optimization 
function”  or  “policy”. 
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Figure  7-11:  General  Playbook  Architecture. 


In  the  remainder  of  this  section,  I  will  describe  two  delegation  architectures  we  are  developing.  While  neither 
system  enables  all  of  the  types  of  delegation  described  above,  and  neither  is  fully  implemented  yet,  collectively 
they  illustrate  the  five  types  of  delegation  and  provide  a  rich  and  highly  flexible  set  of  interactions  for  human- 
automation  delegation. 

7.6.2.1  Playbook  -  Delegation  of  Goals,  Plans  and  Constraints 

The  first  architecture  is  based  on  the  metaphor  of  a  sports  team’s  playbook.  A  playbook  works  because  it 
provides  for  rapid  communication  about  goals  and  plans  between  a  supervisor  (e.g.,  a  coach)  and  a  group  of 
intelligent  actors  (the  players)  who  are  given  the  authority  to  determine  how  to  act  within  the  constraints 
inherent  in  the  coach’s  play.  Our  Playbook  architecture  supports  delegation  action  types  1  -  4  in  principle  and 
has  been  implemented  in  prior  prototypes  to  include  action  types  2  and  4. 

The  basic  Playbook  system  architecture  is  presented  in  Figure  7-8.  The  Playbook  ‘proper’  consists  of  a  User 
Interface  (UI)  and  a  constraint  propagation  planner  known  as  the  Mission  Analysis  Component  (MAC) 
that  communicate  with  each  other  and  with  the  operator  via  a  Shared  Task  Model.  The  operator  communicates 
instructions  in  the  form  of  desired  goals,  tasks,  partial  plans  or  constraints,  via  the  UI,  using  the  task  structures 
of  the  shared  task  model.  The  MAC  is  an  automated  planning  system  that  understands  these  instructions  and 
(a)  evaluates  them  for  feasibility  and/or  (b)  expands  them  to  produce  fully  executable  plans.  The  MAC  may 
draw  on  special  purpose  planning  tools  (e.g.,  an  optimizing  path  planner)  to  perform  these  functions,  wrapping 
them  in  its  task-sensitive  environment.  Outside  of  the  tasking  interface,  but  essential  to  its  use,  are  two 
additional  components.  An  Event  Handling  component,  itself  a  reactive  planning  system  capable  of  making 
momentary  adjustments  during  execution,  takes  plans  from  the  Playbook.  These  instructions  are  sent  to 
control  algorithms  that  actually  effect  behaviors. 

Operator  interaction  with  the  Playbook  can  be  via  a  variety  of  user  interfaces  customized  to  the  needs  of  the 
domain  and  work  environment,  but  operator  commands  are  ultimately  interpreted  in  terms  of  the  Shared  Task 
Model.  To  date,  we  have  developed  prototype  playbooks  for  UCAV  teams  [2],  and  Tactical  Mobile  Robots 
[3],  and  are  currently  developing  prototypes  for  the  RoboFlag  game  [4]  and  for  real-time  interaction  with 
teams  of  heterogeneous  UMVs.  Below,  we  provide  a  description  of  user  interaction  with  one  Playbook 
interface  we  developed  with  Honeywell  Laboratories  to  illustrate  the  general  concept. 
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We  developed  the  playbook  illustrated  in  Figure  7-12  to  enable  a  human  leader  to  create  a  full  or  partial 
mission  plan  for  UCAVs.  This  initial  work  was  intended  as  a  ground-based  tasking  interface  to  be  used  for  a 
priori  mission  planning,  but  current  playbook  work  is  exploring  interface  modifications  to  enable  real-time 
and  in-flight  tasking  and  task  performance  monitoring  as  well. 


Figure  7-12:  Prototype  Playbook  Interface  for  UCAV  Mission  Planning. 


Figure  7-12  shows  five  primary  regions  of  this  Playbook  UI.  The  upper  half  of  the  screen  is  a  Mission 
Composition  Space  that  shows  the  plan  composed  thus  far.  In  this  area,  the  operator  can  directly  manipulate 
the  tasks  and  constraints  in  the  plan.  The  lower  left  corner  of  the  interface  is  an  Available  Resource  Space, 
currently  presenting  the  set  of  aircraft  available  for  use.  The  lower  right  corner  contains  an  interactive  Terrain 
Map  of  the  area  of  interest,  used  to  facilitate  interactions  with  significant  geographic  information  content. 
The  space  between  these  two  lower  windows  (empty  at  startup)  is  a  Resource  in  Use  Space  -  once  resources 
(e.g.,  UCAVs,  munitions,  etc.)  are  selected  for  use,  they  will  be  moved  here  where  they  can  be  interacted  with 
in  more  detail.  Finally,  the  lower  set  of  control  buttons  is  always  present  for  interaction.  This  includes  options 
such  as  “Finish  Plan”  for  handing  the  partial  plan  off  to  the  MAC  for  completion  and/or  review  and  “Show 
Schedule”  for  obtaining  a  Gantt  chart  timeline  of  the  activities  planned  for  each  actor,  etc. 
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At  startup,  the  Mission  Composition  Space  presents  the  three  top-level  plays  (or  ‘mission  types’)  the  system 
currently  knows  about:  Interdiction,  Airfield  Denial,  and  Suppress  Enemy  Air  Defences  (SEAD).  The  mission 
leader  would  interact  with  the  playbook  to,  first;  declare  that  the  overall  mission  “play”  for  the  day  was,  say, 
“Airfield  Denial.”  In  principle,  the  user  could  define  a  new  top-level  play  either  by  reference  to  existing  play 
structures  or  completely  from  scratch,  but  this  capability  has  not  been  implemented  yet. 

This  action  is  an  example  of  type  2  delegation  -  providing  a  specific  task  for  subordinates  to  perform  - 
but  because  this  is  a  very  high  level  task  in  a  hierarchical  task  network,  the  supervisor  has  left  a  great  deal  of 
freedom  to  the  subordinates  (in  this  case,  the  MAC  and  the  UAVs  themselves)  to  determine  exactly  how  an 
“Airfield  Denial”  mission  is  to  be  performed.  If  this  were  the  only  delegation  information  the  supervisor 
provided,  the  subordinates  would  be  obligated  to  do  their  best  to  perform  that  action  (an  Airfield  Denial 
mission),  but  would  have  a  great  deal  of  authority  as  to  how  best  to  accomplish  it. 

At  this  point,  having  been  told  only  that  the  task  for  the  day  is  “Airfield  Denial,”  a  team  of  trained  pilots  would 
have  a  very  good  general  picture  of  the  mission  they  would  fly.  Similarly,  the  tasking  interface  (via  the  Shared 
Task  Model)  knows  that  a  typical  airfield  denial  plan  consists  of  ingress,  attack  and  egress  phases  and  that  it  may 
also  contain  a  suppress  air  defence  task  before  or  in  parallel  with  the  attack  task  -  but  just  as  a  leader  instructing 
a  human  flight  team  could  not  leave  the  delegation  instructions  at  a  simple  ‘Let’s  do  an  Airfield  Denial  mission 
today,’  so  the  operator  of  the  tasking  interface  is  required  to  provide  more  information.  Here,  the  human  must 
provide  four  additional  items:  a  target,  a  homebase,  a  staging  and  a  rendezvous  point.  Each  of  these  is  a 
stipulation ,  or  positive  constraint,  telling  the  subordinates  that  whatever  specific  plan  they  come  up  with  to 
accomplish  the  higher  level  mission  must  include  these  attributes  -  and  thus,  they  are  examples  of  type  4 
delegation  interactions.  Most  of  these  activities  are  geographical  in  nature  and  users  typically  find  it  easier  to 
perform  them  with  reference  to  a  terrain  map.  Hence,  by  selecting  any  of  them  from  the  pop  up  menu,  the  user 
enables  direct  interaction  with  the  Terrain  Map  to  designate  an  appropriate  point.  Since  the  Playbook  knows 
what  task  and  parameter  the  point  is  meant  to  indicate,  appropriate  semantics  are  preserved  between  user  and 
system.  As  for  all  plans,  the  specific  aircraft  to  be  used  may  be  selected  by  the  user  or  left  to  the  MAC.  If  the 
user  wishes  to  make  the  selection,  s/he  views  available  aircraft  in  the  Available  Resource  Space  and  chooses 
them  by  clicking  and  moving  them  to  the  Resources  in  Use  Area. 

The  mission  leader  working  with  a  team  of  human  pilots  could,  if  time,  mission  complexity  or  degree  of  trust 
made  it  desirable,  hand  the  mission  planning  task  off  to  the  team  members  at  this  point.  The  playbook 
operator  can  do  this  as  well,  handing  the  task  to  the  MAC  via  the  “Finish  Plan”  button.  The  leader  might  wish, 
however,  to  provide  substantially  more  detailed  delegation  instructions.  S/he  can  do  this  by  progressively 
interacting  with  the  playbook  UI  to  provide  deeper  layers  of  task  selection,  or  to  impose  more  stipulations  on 
the  resources  to  be  used,  waypoints  to  be  flown,  etc.  For  example,  clicking  on  “Airfield  Denial”  produces  a 
pop-up  menu  with  options  for  the  user  to  tell  the  MAC  to  “Plan  this  Task”  (that  is,  develop  a  plan  to 
accomplish  it)  or  indicate  that  s/he  will  'Choose  airfield  denial’  as  a  task  that  s/he  will  flesh  out  further. 
The  pop-up  menu  also  contains  a  context-sensitive  list  of  optional  subtasks  that  the  operator  can  choose  to 
include  under  this  task.  This  list  is  generated  by  the  MAC  with  reference  to  the  existing  play  structures  in  the 
play  library,  filtered  for  current  feasibility. 

After  the  user  chooses  ‘Airfield  Denial’  the  system  knows,  via  the  Shared  Task  Model,  that  this  task  must 
include  an  Ingress  subtask  (as  illustrated  in  Figure  7-12).  The  supervisor  does  not  have  to  tell  intelligent 
subordinates  this;  it  is  a  part  of  their  shared  knowledge  of  what  an  ‘Airfield  Denial’  task  means  -  and  how  it 
must  be  performed.  To  provide  detailed  instructions  about  how  to  perform  the  Ingress  task,  however,  the  user 
can  choose  it,  producing  a  “generic”  Ingress  task  template  or  “play”.  This  is  not  a  default  method  of  doing 
“Ingress”,  but  a  generic,  uninstantiated  template  -  corresponding  to  what  a  human  expert  knows  about  what 
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constitutes  an  Ingress  task  and  how  it  can  or  should  be  performed.  A  trained  pilot  knows  that  Ingress  can  be 
done  either  in  formation  or  in  dispersed  mode  and,  in  either  case,  must  involve  a  “Take  Off’  subtask  followed 
by  one  or  more  “Fly  to  Location”  subtasks.  Similarly,  the  user  can  select  from  available  options  (e.g.,  formation 
vs.  dispersed  Ingress,  altitude  constraints  on  takeoff,  etc.)  on  context-sensitive,  MAC-generated  menus 
appropriate  to  each  level  of  decomposition  of  the  task  model.  One  of  our  current  challenges  in  creating 
Playbooks  for  real-time  interactions  is  to  enable  them  to  be  sensitive  to  the  current  state  of  affairs  and  of  task 
performance  so  as  to  make  intelligent  assumptions  about  task  performance  possible  -  for  example,  if  the 
supervisor  wishes  to  command  a  currently  airborne  UAV,  perhaps  in  a  holding  pattern,  to  perform  an  ‘Airfield 
Denial’  mission,  both  supervisor  and  subordinate  should  know  that  the  Takeoff  portion  of  an  Ingress  task  is  no 
longer  necessary  and  should  either  be  eliminated  or  be  shown  as  already  accomplished. 

The  user  can  continue  to  specify  and  instantiate  tasks  down  to  the  “primitive”  level  where  the  sub-tasks  are 
behaviors  the  control  algorithms  (see  Figure  7-1 1)  on  board  the  aircraft  can  be  relied  upon  to  execute  in  flight. 
Alternatively,  at  any  point  after  the  initial  selection  of  the  top  level  mission  task  and  its  required  parameters, 
the  supervisor  can  hand  the  partly  developed  plan  over  to  the  MAC  for  completion  and/or  review.  In  extreme 
cases,  a  viable  “Airfield  Denial”  plan  for  multiple  aircraft  could  be  created  in  our  prototype  with  as  few  as 
five  selections  and  more  sophisticated  planning  capabilities  could  readily  reduce  this  number  further  - 
but  potentially  more  important,  the  operator  (like  a  human  supervisor  dealing  with  intelligent  subordinates) 
can  also  provide  more  detailed  instructions  whenever  s/he  deems  them  necessary  or  useful  to  performing  the 
mission  successfully  and  in  the  way  s/he  sees  fit. 

This  Playbook  illustrates  delegation  interactions  2  and  4  (plans  and  stipulations).  The  subordinates’  role  in  these 
types  of  interaction  are  described  in  the  table  above  -  to  perform  the  plan  through  any  set  of  sub-methods  that 
adhere  to  the  stipulations  provided  by  the  supervisor,  or  to  report  that  this  is  infeasible.  One  of  the  MAC’S  roles 
in  the  above  example  is  to  report  when  it  is  incapable  of  developing  a  viable  plan  within  the  constraints 
imposed,  (e.g.,  if  the  user  has  stipulated  distant  targets  that  exceed  aircraft  fuel  supplies).  In  a  real-time 
delegation  system,  the  MAC  will  be  responsible  for  continual  monitoring  of  performance  to  report  when 
world  states  mean  that  plan  performance  is  no  longer  capable  of  (or  likely  to)  accomplish  the  user’s  parent 
plan  (e.g.,  because  of  equipment  failures,  adverse  head  winds,  enemy  countermeasures,  etc.) 

The  Playbook  architecture  is,  we  believe,  also  capable  of  supporting  delegation  interaction  types  1  and  3 
(goals  and  negative  constraints)  as  well.  Supporting  goal-based  delegation  interactions  would  require  a  slight 
modification  to  the  shared  task  representation.  Currently,  we  have  used  a  representation  that  explicitly 
includes  only  hierarchically  organized  and  sequenced  tasks  (i.e.,  actions  to  be  performed).  Tasks  implicitly 
encode  the  goals  they  accomplish,  but  there  are  representations  (such  as  Geddes  Plan-Goal  Graphs  [5]) 
that  explicitly  interleave  both  plans  and  goals  and  a  linked  hierarchy.  Use  of  such  a  representation,  along  with 
related  modifications  to  the  UI  and  MAC,  would  enable  the  supervisor  to  say,  effectively,  “Today  we’re  going 
to  achieve  a  State”  (e.g.,  the  destruction  of  a  given  airfield)  rather  than  or  in  addition  to,  the  plan-based 
representation  used  above  which  allows  only  the  issuing  of  task-based  delegation  commands  (e.g.,  “Today 
we’re  going  to  fly  an  airfield-denial  mission”).  The  incorporation  of  negative  constraints  into  the  interaction 
(delegation  interaction  method  #3),  would  require  a  less  substantial  modification  to  the  Playbook  architecture 
-  potentially  requiring  only  a  UI  addition  to  enable  the  supervisor  to  incorporate  negative  commands  about 
task  types  and  state  parameters  (e.g.,  “do  NOT  fly  through  this  valley  or  use  this  type  of  munition”)  and  then 
requiring  the  MAC  to  create  plans  which  avoid  those  negative  constraints. 

7 .6.2.2  Policy  -  Delegation  via  Abstract  Value  Statements 

The  final  type  of  delegation  interaction  offers  the  ability  to  provide  priorities  between  alternate  goals  and 
states  and  to  do  so  more  abstractly  than  the  above  methods.  Sometimes  supervisors  don’t  have  a  single, 
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concrete  world  state  goal  in  mind,  much  less  a  specific  plan  for  accomplishing  it.  Sometimes  supervisors  must 
issue  commands  well  in  advance  to  cover  a  wide  range  of  largely  unanticipatable  circumstances.  In  these 
cases,  the  delegation  instructions  will  be  less  a  specific  statement  of  actions  to  take  or  world  states  to  be 
sought  or  avoided,  but  rather  a  general  statement  of  outcomes  that  would  be  more  or  less  good  or  valuable 
(or,  conversely,  bad  or  to  be  avoided)  than  others.  We  refer  to  the  set  of  such  abstract  value  statements  that  a 
supervisor  might  provide  as  his  or  her  “policy”  for  performance  in  the  domain. 

We  have  developed  policy-based  architectures  for  two  applications:  providing  commanders’  guidance  to  a 
resource  controller  for  battlefield  network  communications  [6],  and  providing  visualization  and  feedback  to 
dispatchers  in  upset  contexts  in  commercial  aviation  [7].  We  will  describe  the  first  of  these  below. 

A  policy  statement  is  an  abstract,  general,  a  priori  statement  of  the  relative  importance  or  value  of  a  goal  state 
in  the  domain.  In  its  simplest  form,  policy  provides  a  method  for  human  operators  to  mathematically  define 
what  constitutes  “goodness”  in  terms  of  the  outcomes  of  the  delegation.  Once  defined,  a  policy  statement  can 
be  treated  as  a  rule  and  evaluated  against  a  current  or  hypothetical  context  -  if  the  rule  is  true  in  the  context, 
then  the  context  incurs  the  “goodness”  (or  badness)  value  stipulated  by  the  rule.  Alternate  contexts 
(which  could  be  tied  to  the  expected  outcomes  of  alternate  decisions)  can  then  be  evaluated  against  each  other 
by  examining  the  set  of  policy  rules  that  are  satisfied  or  violated  and  the  resulting  set  of  goodness/badness 
values  accrued.  A  set  of  individual  policy  statements  can  be  bundled  together,  and  these  policy  bundles  can  be 
used  to  flexibly  define  the  priorities  that  apply  in  a  given  situation  (priorities  can  change  given  different 
circumstances). 

A  policy-based  delegation  system  requires  at  least  three  components: 

1)  A  representation  for  specifying  the  “policy”  in  terms  of  the  value  of  various  partial  world  states; 

2)  A  user  interface  for  allowing  one  or  more  users  to  input  their  policies  and,  if  desired,  view  results  of 
policy  application; 

3)  A  computational  framework  that  allows  evaluation  of  a  current  situation  or  hypothetical  proposed 
situation  against  the  expressed  policy;  or 

4)  An  engine  that  allows  application  of  the  policy  either  to  the  control  of  resource  application  or  to  a 
visualization  of  sensed  data  about  a  current  situation  or  projected  data  about  a  future  or  simulated 
situation. 

In  the  development  of  a  policy-based  delegation  system  for  communications  resource  usage,  we  were  striving 
to  provide  a  means  for  commanders  to  tell  an  automated  network  management  system  their  “policies”  for  how 
to  prioritize  the  use  of  communications  bandwidth  in  order  to  satisfy  the  most  important  requests  most  fully. 
Note,  however,  that  “most  important”  was  not  a  static  concept,  but  rather  changed  across  commanders  and 
situations.  For  this  application,  we  developed  a  policy  representation  that  allowed  commanders  to  assign, 
a  priori  and  abstractly,  a  value  to  various  kinds  of  communications  requests.  As  communications  requests  then 
came  in  from  various  field  units  or  operators,  they  could  be  matched  against  the  commander’s  policy 
statements  and  a  value  assigned  to  each  of  them.  This  value  was  then  used  by  a  resource  optimizing  controller 
to  determine  which  requests  should  get  network  bandwidth  with  what  priority. 

This  process  is  conceptually  illustrated  in  Figure  7-13.  Each  commander’s  policy  is  created  as  a  set  of  statements 
each  of  which  assigns  an  importance  (or  value)  function  to  a  defined  sub-region  in  a  multidimensional  space. 
Regions  might  be  based  on  a  single  dimension  (‘Requests  for  weather  information  [Content]  get  Importance 
0.2’)  or  on  a  combination  of  dimensions  (‘Requests  owned  by  the  Zone  Reconnaissance  task  [Owner] 
for  weather  information  [Content]  from  Satellite  476B  [Source]  to  3rd  Air  Calvary  Division  [Destination] 
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get  Importance  0.8).  If  the  policy  element  regions  are  allowed  to  overlap,  then  they  must  be  sequenced  (typically 
from  most  to  least  specific)  to  indicate  the  order  of  precedence.  In  practice,  the  commander’s  policy  is  then  used 
to  assign  an  importance  value  to  any  incoming  request  for  communication  resources  (illustrated  conceptually  in 
Figure  7-13).  Each  incoming  request  is  matched  against  the  sequenced  series  of  policy  element  statements  the 
commander  has  made.  The  first  policy  element  that  matches  the  request  determines  the  importance  of  that 
request  and  informs  an  automated  resource  manager  about  the  relative  value  of  satisfying  that  request. 
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Figure  7-13:  Representation  of  a  Policy  for  Network  Bandwidth  Prioritization. 


While  conceptually  simple,  many  useful  functions  can  be  performed  within  this  framework.  First,  it  is  not 
necessary  that  importance  be  constmed  as  an  all-or-nothing  value  as  it  is  depicted  in  7-13.  Instead,  we  have 
explored  more  sophisticated  representations  that  allow  the  requestor  to  provide  a  description  of  how  s/he 
wants  the  information  requested  along  several  dimensions  (e.g.,  freshness,  reliability,  initiation-time, 
accuracy,  resolution,  scope,  etc.)  Then  the  resource  management  system  can  treat  the  importance  value  as  a 
maximum  number  of  value  “points”  to  be  awarded  for  satisfying  the  request  perfectly,  while  still  awarding 
itself  points  for  partial  satisfaction.  This  permits  more  sensitive  management  of  resources  to  be  performed. 

Second,  it  is  rarely  the  case  that  a  single  commander  or  supervisor  is  the  only  one  who  may  have  an  interest  in 
dictating  policy  about  how  subordinates  behave.  Rather,  each  commander  must  allocate  his/her  resources  in 
accordance  with  the  policies  of  those  above.  We  support  this  requirement  (Figure  7-14)  by  modeling  policies 
that  exist  at  nodes  in  a  command  hierarchy.  As  requests  come  in,  they  are  matched  against  the  commander’s 
policy  that  governs  them,  but  must  then  also  be  matched  against  his/her  commander’s  policy  -  and  so  on, 
up  the  chain  of  command.  We  allow  each  commander  to  stipulate  how  this  matching  policy  element  should  be 
resolved  with  the  subordinate  commander’s  matching  policy  element:  as  a  ceiling  or  floor  value,  or  linear 
combination  of  the  values.  Even  when  a  single  well-defined  chain  of  command  does  not  exist  the  policies  of 
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different  “interest  groups”  may  be  represented  with  relative  weights  on  the  importance  values  that  each  would 
assign  to  a  potential  outcome.  We  used  this  approach  in  representing  the  many,  varied  interests  which  impact 
the  decisions  of  a  commercial  airline’s  dispatch  operators  (e.g.,  crew  scheduling,  maintenance  scheduling, 
marketing,  passengers,  finance,  etc.)  [7]. 


Figure  7-14:  Cross  Echelon  Policy  Application  and  Resolution. 

Folding  this  policy-based  form  of  delegation  interaction  (method  5)  into  an  overall  architecture  that  includes 
the  other  methods  is  not  as  difficult  as  it  might  first  appear.  While  we  have  not  yet  developed  a  system  that 
accomplishes  this,  the  way  forward  is  clear.  Policy  is  simply  an  assignment  of  value  or  priority  to  the  goal 
states  and  tasks  in  the  other  delegation  interaction  types.  Priorities  for  resource  usage  and  the  desirability  of 
various  outcomes  stem,  after  all,  from  a  superior’s  goals  and  plans  for  subordinates  (whether  human  or 
machine).  If,  for  example,  I  task  a  given  unit  under  my  command  to  perform  an  Airfield  Denial  task,  and 
I  know  that  their  task  is  the  most  important  of  all  concurrent  tasks  to  me,  then  I  have  effectively  said  that 
giving  them  the  resources  they  require  to  perform  that  task  (specific  aircraft,  munitions,  fuel,  communications 
bandwidth,  etc.)  represents  the  highest  value  to  me.  In  other  words,  delegation  interactions  that  provide 
specific  goals,  plans,  stipulations  and  constraints  to  subordinates  carry  with  them  specific  policy  implications. 
Whenever  a  commander  can  provide  more  specific  delegation  instructions,  this  will  generally  get  him/her 
closer  to  the  results  desired  from  his/her  subordinates,  but  this  will  not  always  be  the  case.  Hence,  the  ability 
to  stipulate  more  abstract  policies  should  probably  be  preserved  in  a  complete  delegation  system  as  a  means  of 
covering  unexpected  and  unfamiliar  situations. 

7.6.3  Conclusions  and  Future  Work 

While  the  work  described  above  represents  a  general  framework  for  delegation  interactions  suitable  for  human 
interaction  with  smart  automation  of  various  kinds  and,  perhaps  uniquely,  suitable  for  the  tasking  of  multiple 
UMVs,  our  work  has  thus  far  progressed  only  to  the  proof  of  concept  stage.  As  noted  above,  we  have 
currently  implemented  only  portions  of  the  various  methods  of  delegation  that  a  fully  flexibly  delegation 
interface  might  benefit  from,  and  have  done  so  in  disparate  systems.  Furthermore,  our  proof  of  concept 
implementations  have  not  yet  afforded  us  the  opportunity  to  do  rigorous  human  in  the  loop  evaluations  to 
demonstrate  improved  performance,  if  any. 
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These  situations  are  changing,  however.  We  are  currently  engaged  in  exploration  of  human  interaction  with 
Playbook-like  interfaces  with  Parasuraman  and  Galster  [4]  and  are  beginning  work  on  a  Playbook  interface  for 
real-time  interactions  with  heterogeneous  UMV  assets  by  operators  who  may  be  concurrently  involved  in 
other  critical  tasks  (under  a  DARPA-IXO  SBIR  grant).  One  of  the  goals  of  this  work  will  be  to  develop  task 
libraries  and  task  construction  tools  and  interface  concepts  to  move  the  delegation  interface  work  along 
toward  implementation  and  utility. 

Of  course,  anyone  who  has  worked  with  a  poorly  trained,  or  simply  mismatched,  subordinate  is  well  aware 
that  it  is  possible  for  delegation  to  cause  more  work  than  it  saves.  Our  challenge,  and  that  of  others  who  adopt 
a  delegation  framework  for  human  interaction  with  complex  and  largely  autonomous  automation,  will  be  to 
ensure  that  this  does  not  happen  -  through  judicious  use  of  technology  and  substantial  usability  analysis  and 
testing.  On  the  positive  side,  however,  we  benefit  from  the  knowledge  that  delegation  approaches  to 
interaction  with  intelligent  yet  subordinate  actors  have  worked  repeatedly  throughout  history  and,  particularly, 
the  history  of  warfare.  As  automation  in  the  form  of  UMVs  increasingly  takes  its  place  as  one  of  those  actors 
we  want  to  be  intelligent,  capable  and  effective  yet  remain  subordinate,  we  will  increasingly  need  methods  for 
enabling  it  to  interact  with  us  in  the  ways  that  we  trust  and  are  familiar  with.  Since  delegation  is  the  primary 
method  we  have  evolved  to  meet  these  requirements,  it  only  makes  sense  to  pursue  delegation  approaches  to 
human  interaction  with  automation. 
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7.7  MODELLING  MULTI-LAYERED  CONTROL:  APPLICATION  OF  THE 

EXTENDED  CONTROL  MODEL  TO  THE  ANALYSIS  OF  UAV  SCENARIOS 

Automation  has  often  been  approached  from  the  bottom  up,  starting  with  the  system  components.  This  has 
focused  the  issues  on  the  merits  of  humans  and  machines  either  as  a  comparison  of  attributes  and  capabilities, 
or  in  terms  of  relative  roles  as  in  levels  of  automation.  An  alternative  is  to  approach  the  problem  from  the  top 
down,  using  the  requirements  to  joint  system  performance  as  a  starting  point.  In  this  approach  the  emphasis  is  on 
being  in  control.  The  controlling  system  must  match  the  requisite  variety  of  processes  involving  subsystems  with 
different  dynamics,  degrees  of  complexity,  and  predictability.  This  requires  multiple,  simultaneous  layers  of 
control.  A  multi-layered  control  model  provides  a  good  basis  for  understanding  the  consequences  of  automation 
and  the  needs  of  various  types  of  information  to  support  views  of  the  past,  present,  and  future.  The  top  down 
approach  is  illustrated  by  describing  the  four  layers  needed  to  control  a  vehicle. 

The  objective  of  this  note  is  to  demonstrate  how  an  UAV  scenario  can  be  described  using  the  principles  of  the 
Extended  Control  Model  (ECOM).  This  in  turn  requires  a  discussion  of  the  Autonomous  Control  Level  (ACL) 
framework,  which  provides  a  common  reference  for  work  on  UAVs  relating  to  issues  of  automation  and 
human  factors.  This  report  comprises  three  parts:  a  brief  introduction  to  the  ECOM,  a  discussion  of  the  ACL 
framework,  and  a  description  of  an  UAV  scenario  using  the  ECOM  as  a  frame  of  reference. 

7.7.1  Introduction 

This  chapter  considers  how  to  approach  the  problem  of  controlling  one  or  more  UAVs2  to  accomplish  a  given 
mission.  This  is  a  problem  that  is  found  in  military  as  well  as  civilian  domains,  although  it  is  the  former  that 
provides  the  context  here.  The  issue  is  basically  how  to  define  and  achieve  an  effective  balance  between 
manual  and  automatic  control,  i.e.,  between  having  an  operator  guide  the  vehicle  and  having  the  vehicle  guide 
itself.  This  issue  brings  to  the  fore  many  of  the  problems  that  have  been  dealt  with  in  the  field  of  industrial 
automation,  but  also  adds  new  perspectives  and  demands. 

The  chapter  first  presents  a  general  framework  for  how  to  describe  this  problem,  based  on  the  principles  of 
cognitive  systems  engineering.  It  then  illustrates  the  principles  of  the  description  by  considering  an  example 
taken  from  the  military  domain.  The  emphasis  of  the  chapter  is  to  present  a  methodological  approach  that  can 
be  used  to  make  the  problems  tractable  and,  hopefully,  solvable. 

7.7.1. 1  Control  as  a  Cognitive  Engineering  Problem 

With  the  possible  exception  of  a  few,  unique  cases  the  practical  need  to  control  something  -  a  process, 
a  vehicle,  or  a  socio-technical  system  -  always  faces  two  predicaments.  One  is  that  control  invariably  takes 
place  in  time  and  requires  time:  processes  are  dynamic  and  developments  are  only  incompletely  predictable. 
The  other  is  that  usually  more  than  one  activity  is  going  on,  which  means  that  more  than  one  line  of 
development  must  be  kept  keep  track  of.  Even  in  cases  where  a  controller  can  focus  on  a  single  process, 


2 

The  term  UAV  -  for  uninhabited  aerial  (or  airborne)  vehicle  -  is  used  as  a  general  reference  to  unmanned  vehicles,  including  land- 
based  and  sea-based  versions. 
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that  process  may  have  to  be  considered  with  multiple  time-horizons,  hence  effectively  as  being  composed  of 
multiple  and  potentially  asynchronous  processes.2 3 

The  consequence  of  this  is  that  it  is  inadequate  to  address  the  control  problem  by  methods  that  use 
decomposition  as  the  main  principle.  Any  system  can  structurally  be  described  as  being  composed  of  a 
number  of  parts  or  subsystems,  which  in  turn  may  be  described  as  composed  of  parts  or  subsystems  and  so  on, 
until  the  level  of  elementary  components  is  reached.4  Yet  while  such  descriptions  are  highly  suitable  for  an 
instantiation  of  a  system’s  structure,  they  are  not  necessarily  the  best  way  to  describe  a  system’s  functions. 
Here  it  may  be  more  appropriate  to  begin  by  understanding  what  systems  do  -  or  what  they  are  supposed  to  do 
-  rather  than  by  describing  what  they  are  made  of.  Decomposition  is  a  powerful  analytical  tool,  which  has 
become  an  almost  sacred  principle  of  Western  science.  Because  of  that  it  is  too  often  taken  for  granted,  which 
means  that  it  sometimes  is  used  when  it  should  not  have  been.  For  instance,  if  we  apply  the  notion  of  a 
human-machine  system  this  already  embraces  the  assumption  that  it  is  meaningful  to  make  a  distinction 
between  humans  and  machines  as  two  major  groups  of  components.  This  assumption  can,  however,  easily  be 
questioned.  It  is,  indeed,  entirely  possible  to  start  an  analysis  from  the  notion  of  goals  and  means  and  use  that 
as  the  guiding  principle  when  a  system  description  is  developed.  This  is  a  well-established  tradition  as 
demonstrated  by  the  classical  work  of  Miller,  Galanter  and  Pribram  [1]  although  it  often  is  disregarded  by  the 
mainstream  of  HCI  and  HMI.5 

Discussions  of  how  humans  and  machines  can  work  together  to  the  benefit  of  the  overall  human-machine 
system  can,  of  course,  not  avoid  the  thorny  issue  of  how  functions  should  be  allocated  or  distributed  between 
the  parts.  Even  if  the  starting  point  is  a  functional  analysis,  it  is  sooner  or  later  necessary  to  consider  how  the 
resulting  design  should  be  implemented.  Function  allocation  also  implies  that  a  certain  level  of  automation 
exists,  since  without  that  machines  would  be  incapable  of  functioning  independently,  hence  to  have  functions 
assigned  to  them.  Indeed,  the  word  automation,  which  is  a  combination  of  the  Greek  words  auto-,  “self’  and 
- matos ,  “willing”,  means  something  that  acts  by  itself  or  of  itself.  In  the  modem  usage  an  artefact  is  said  to  be 
automatic  if  it  is  self-regulating  or  able  to  act  or  operate  in  a  manner  that  is  determined  by  the  conditions, 
but  which  is  essentially  independent  of  external  control. 

To  illustrate  the  differences  in  how  humans  and  machines  can  work  together,  consider  an  automobile  from  the 
1960s,  such  as  an  Austin  Mini  or  a  Volkswagen.  In  these  cars  very  few  things  were  automated  -  indeed, 
even  the  windshield  wipers  had  to  be  stopped  manually  in  the  right  position.  For  such  cars  there  was  therefore 
no  need  to  consider  function  allocation  or  to  worry  about  the  relation  between  humans  and  automation. 
The  driver  had  to  do  everything,  specifically  to  control  speed  and  direction  while  maintaining  sufficient  safety 
margins.  The  situation  is  completely  different  for  a  modem  car.  At  the  top  of  the  range,  cars  are  typically 
equipped  with  systems  such  as  cmise  control  -  or  even  adaptive  cmise  control,  anti-locking  brakes,  electronic 
stability  programs,  traction  control,  and  lane  departure  warnings  as  well  a  electronic  climate  control, 
automatic  transmission,  rain  sensors,  etc.  The  driver  still  has  to  control  the  direction  and  speed  of  the  car  - 
or  at  least  to  indicate  the  desired  direction  and  speed,  since  in  many  cases  the  car  can  then  take  care  of  the  rest. 
In  short,  the  situation  has  changed  completely,  and  the  control  needed  to  ensure  safe  driving  is  now  distributed 
between  the  driver  and  a  number  of  automated  systems  in  the  vehicle.6 


2 

It  might  be  more  correct  to  say  that  the  operator  focuses  on  only  one  goal.  Any  goal  may  be  considered  from  different  perspectives 
and  using  different  criteria,  hence  require  multiple  processes  to  be  achieved. 

4  As  the  story  of  the  atom  shows,  the  level  of  elementary  components  is  never  absolute,  but  depends  on  the  frame  of  reference. 

5  The  basic  ‘component’  in  this  work  was  the  Test-Operate-Test-Exit  (TOTE)  unit,  which  describes  a  pattern  of  activity  rather  than  a 
structure. 

6  Although  the  gradual  transition  of  control  from  the  driver  to  the  vehicle  always  is  done  with  the  best  of  intentions,  it  sooner  or  later 
leads  to  unexpected  problems  of  risk  and  responsibility. 
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From  a  historical  perspective,  automation  has  been  used  either  to  ensure  a  more  precise  performance  of  a 
given  function  or  to  improve  the  stability  of  system  performance.  After  the  industrial  revolution,  a  further 
motivation  was  to  increase  the  speed  of  work  and  production.  The  net  effect  of  the  increasing  use  of 
technology  was  that  humans  gradually  came  to  be  seen  as  a  bottleneck  for  system  performance,  thereby 
strengthening  the  need  of  further  automation.  It  is  no  coincidence  that  human  factors  engineering  emerged  in 
the  late  1940s,  when  the  rapid  changes  brought  about  by  the  scientific  and  technological  developments  in  the 
preceding  decade  had  undermined  the  hitherto  peaceful  co-existence  between  humans  and  machines.  These 
developments  opened  the  field  for  a  new  type  of  engineers,  branded  rather  appropriately  by  Norbert  Wiener  as 
gadget  worshippers  [2,  p.  53],  defined  as  people  who  “regard(ed)  with  impatience  the  limitations  of  mankind, 
and  in  particular  the  limitation  consisting  in  man’s  undependability  and  unpredictability.”  In  addition  to  being 
seen  as  a  bottleneck  for  system  performance  humans  were  also  perceived  as  a  source  of  unwanted  variability 
that  was  a  primary  cause  of  incidents  and  accidents.  According  to  the  logic  of  this  line  of  reasoning, 
it  therefore  made  sense  to  replace  humans  by  automation  as  far  as  possible.  Unfortunately,  this  view  has 
created  many  of  the  problems  we  face  today. 

7.7.1.2  Bottom-Up  Function  Allocation 

As  soon  as  function  allocation  was  recognised  to  be  a  problem,  methods  or  principles  were  proposed  to  solve 
it.  Most  of  these  started  from  the  bottom  up,  i.e.,  from  the  components  and  functions  that  were  seen  as 
constituting  the  system.  Humans  and  machines  were  regarded  as  two  separate  components  with  distinct 
functions,  which  in  the  case  of  humans  typically  were  sensory-motor  or  mental  (cognitive)  functions.  In  a 
similar  manner  tasks  were  decomposed  into  subtasks  and  activities,  and  a  component’s  ability  to  carry  out 
specific  activities  became  the  basis  for  function  allocation  and  automation.  This  could  be  done  either  by 
focusing  on  the  activities  as  such  [3]  or  by  comparing  humans  and  machines  function  by  function  [4]. 
The  underlying  premise  was  that  humans  were  needed  for  machines  to  work,  and  the  problem  was  to  ensure 
that  this  could  be  done  in  an  effective  manner  -  although  from  a  technological  rather  than  psychological 
perspective.  (If  humans  were  not  needed  for  machines  to  work,  the  problem  would  obviously  disappear. 
This  condition  can,  however,  only  be  achieved  by  fully  automated  systems.) 

The  reasons  for  using  the  bottom-up  approach  are  not  hard  to  find.  Automation  began  by  applying  technology 
to  take  over  relatively  simple  and  well-defined  functions,  such  as  James  Watt’s  Governor  that  controlled  the 
speed  of  the  steam  engine.  As  long  as  technology  was  employed  to  take  care  of  isolated  functions,  it  made 
perfect  sense  to  consider  these  alone  and  to  compare  humans  and  machines  in  terms  of  their  specific 
capabilities.  Once  the  tradition  had  been  successfully  established  it  was,  however,  carried  over  to  cases  where 
it  should  have  been  used  with  greater  care  or  not  have  been  used  at  all.  These  are  typically  the  cases  where  it 
is  incorrect  simply  to  decompose  a  system  into  its  elements. 

Common  to  all  bottom-up  approaches  is  that  they  provide  an  assessment  of  the  system  in  terms  of  component 
characteristics,  rather  than  in  terms  of  the  overall  functioning.  The  best-known  example  is  probably  the  Fitts 
list  [4],  which  compares  the  capabilities  of  humans  and  machines  by  a  fixed  set  of  attributes.  A  more  recent 
example  is  the  scale  used  to  describe  degrees  or  levels  of  automation  [5],  which  proposes  different  roles  or 
areas  of  responsibility  for  humans  and  machines.  Although  the  direct  purpose  is  to  describe  -  and  appraise  - 
the  resulting  system,  rather  than  to  support  function  allocation  the  starting  point  is  nevertheless  the  same: 
humans  and  machines  seen  as  separate  entities  that  can  be  compared  by  means  of  a  common  feature  -  in  this 
case  the  degree  of  autonomy.  The  description  is  thus  one  of  how  much  humans  and  machines  do  relative  to 
each  other,  rather  than  of  what  they  do.  Since  the  various  possible  outcomes  furthermore  are  considered  to 
have  different  values,  this  evaluation  serves  indirectly  as  a  guideline  for  function  allocation.  This  aspect  is 
even  more  noticeable  in  a  later  version  of  the  scale,  cf.  Sheridan  [6]. 
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7.7.1.3  Top-Down  Function  Allocation 

The  alternative  to  a  bottom-up  approach  is  to  begin  from  the  top  down  by  considering  the  requirements  to  the 
performance  of  the  overall  system.  An  example  of  that  is  the  concept  of  critical  functions  [7],  which  has  been 
used  for  design  of  displays  and  operational  support  in  nuclear  power  plants.  While  critical  functions  are 
common  to  all  systems  they  differ  widely  among  domains.  For  instance,  maintaining  lift  is  a  critical  function 
in  aviation,  whereas  in  nuclear  power  production  it  is  essential  to  maintain  cooling  inventory.  Two  well- 
known  examples  of  top-down  analysis  in  the  behavioural  sciences  are  the  General  Problem  Solver  [8]  and  the 
previously  mentioned  Test-Operator-Test-Exit  (TOTE)  operator  [1].  They  both  represent  the  more  general 
principle  of  goals-means  analysis. 

Although  it  is  possible  to  define  a  set  of  general  functions  that  apply  to  any  system  [9],  it  is  sufficient  for  the 
present  discussion  to  consider  just  one  function,  namely  that  of  maintaining  control.  This,  of  course,  requires  a 
clarification  of  “who”  is  in  control  of  “what”.  The  “what”  is  usually  a  dynamic  process,  i.e.,  something  that  is 
capable  of  continuous  and  spontaneous  change,  activity,  or  progress.7  In  a  similar  manner,  the  “who”  refers  to 
the  controller  or  the  controlling  system.  Both  process  and  controller  are  systems,  which  means  that  they  can  be 
described  as  “a  set  of  objects  together  with  relationships  between  the  objects  and  between  their  attributes” 
[10,  p.  81].  Since  humans  are  cognitive  systems  [11,12],  and  since  the  controlling  system  is  assumed  to 
comprise  both  humans  and  technology,  it  is  here  referred  to  as  a  joint  cognitive  system  (JCS). 

To  define  the  meaning  of  being  in  control,  it  is  useful  to  start  by  noting  how  being  out  of  control  generally  is 
associated  with  the  occurrence  of  unwanted  conditions.  Being  in  control  consequently  means  having  the 
power  or  ability  to  direct  and  manage  the  development  of  events,  while  not  being  in  control  means  that  this 
ability  is  temporarily  or  permanently  lost.  A  joint  system  is  defined  as  being  in  control  of  a  situation  either  if 
unexpected  conditions  do  not  arise,  or  if  it  is  possible  to  avoid  unwanted  outcomes  of  such  conditions. 
The  former  means  that  the  joint  system  is  able  to  prevent  unexpected  conditions  from  occurring;  the  latter 
means  that  the  joint  system  is  able  effectively  to  recover  from  such  conditions,  should  they  occur. 

In  the  top-down  view,  the  objective  of  function  allocation  is  to  ensure  that  control  is  maintained.  It  is  therefore 
necessary  to  consider  the  effects  of  function  allocation  -  and  of  automation  -  for  a  variety  of  conditions  over 
time,  rather  than  for  a  pre-defined  set  of  steady  conditions.  It  is  also  necessary  to  consider  which  functions  are 
required  to  maintain  control  and  how  they  depend  upon  and  support  each  other,  rather  than  how  they  compare 
to  each  other  on  specific  attributes  of  humans  and  machines.  The  structurally  based  distinction  between 
system  parts,  such  as  humans  and  machines,  is  in  this  way  replaced  by  a  differentiation  among  the  functions 
needed  to  achieve  the  overall  system  objectives. 

7.7.1.4  Layers  of  Control 

Depending  on  the  domain,  being  in  control  can  mean  to  arrive  at  a  specific  destination  at  a  given  time, 
to  deliver  something  at  a  given  place  -  and  also  at  a  certain  time,  to  keep  the  value  of  selected  process 
parameters  within  a  certain  range,  etc.  For  some  processes,  such  as  heating  a  room  or  maintaining  the  water 
level  in  a  vessel,  control  is  usually  uncomplicated  and  can  be  accomplished  by  a  simple  feedback  loop. 
Most  processes,  however,  comprise  several  functions  or  subsystems.  The  room  may,  for  instance,  be  part  of  a 
larger  building  complex  or  a  special  facility,  and  the  vessel  may  be  a  storage  tank  in  a  power  plant.  In  these 
cases  the  control  of  a  single  function  is  just  part  of  controlling  the  overall  system.  Subsystems  may  have 
different  dynamics  (rate  of  change  or  rate  of  response),  they  may  differ  in  degree  of  complexity  and 


7  It  is  important  to  keep  in  mind  that  the  term  process  can  refer  to  either  a  physical,  a  mental,  or  a  social  process.  The  system  can 
thus  be  a  technological  artefact,  a  human  being,  or  an  organisation. 
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predictability,  and  specific  relations  or  dependencies  may  exist  among  them  (cf.  the  definition  of  a  system 
given  above).  Yet  effective  control  must  be  able  to  consider  all  these  differences. 

This  requirement  is  consistent  with  the  Law  of  Requisite  Variety.  The  simplest,  and  perhaps  best  known, 
formulation  of  this  is  also  the  title  of  a  seminal  paper  on  the  subject:  “Every  good  regulator  of  a  system  must 
be  a  model  of  that  system”  [13].  This  basically  means  that  an  effective  regulator  or  controller  of  a  system  - 
or  a  process  -  must  be  capable  of  responding  to  any  situation  that  can  possibly  occur.  In  relation  to  controlling 
a  vehicle,  for  instance,  it  means  that  the  controller  must  be  able  to  compensate  for  all  disturbances  and  events 
such  that  the  vehicle  is  kept  within  the  envelope  of  safe  and  efficient  performance. 

7.7.1.5  Control  and  Models 

The  Law  of  Requisite  Variety  is  generally  interpreted  to  mean  that  the  controller  must  have  a  sufficiently 
powerful  model  of  the  system  being  controlled.  This  tradition  is  especially  strong  in  the  field  of  human- 
machine  studies,  where  the  operator’s  (mental)  model  has  become  a  sine  qua  non.  The  Law  of  Requisite 
Variety  nevertheless  does  not  say  that  the  regulator  or  controller  must  have  a  model  of  the  system,  but  only 
that  it  must  be  a  model  of  the  system  [14].  The  difference  between  the  two  interpretations  is  important  and  has 
wide-ranging  consequences.  In  the  conventional  way  of  thinking  about  models  the  unspoken  assumption  is 
that  there  must  be  both  a  model  of  what  needs  to  be  known  about  the  process,  and  a  “mechanism”  of  some 
sort  to  execute  or  interpret  the  model.  This  convention  has  been  established  by  artificial  intelligence  and 
knowledge-based  systems  (cf.  the  notion  of  an  inference  engine  in  expert  systems),  and  is  possibly  a  rudiment 
from  Aristotelian  logic.  The  need  to  keep  models  and  mechanisms  separate  and  distinct  is,  however, 
a  convention  rather  than  a  law  of  nature.  In  other  words,  the  variety  that  the  controller  must  have  (to  match  the 
requisite  variety)  need  not  be  represented  explicitly  by  a  model  or  a  formal  representation,  but  can  be  a  feature 
of  how  the  controller  functions  as  such  -  specifically  that  the  controller  can  respond  adequately  to  every 
possible  situation. 

In  the  field  of  human-machine  studies,  systems  that  need  to  be  controlled  are  usually  complex  and  comprise 
multiple  processes  that  develop  with  different  speeds  and  which  therefore  must  be  described  by  different  time 
frames.  In  the  example  of  maintaining  the  water  level  in  a  storage  tank,  the  time  frame  for  that  (sub)process  is 
different  from  the  time  frame  of  power  production.  Filling  or  emptying  a  tank  is  a  matter  of  minutes,  rather 
than  hours  while  power  production  is  a  process  that  develops  more  slowly,  and  which  must  be  considered  in  a 
time  frame  of  hours  or  days.  The  control  of  the  overall  process  (power  production)  thus  differs  from  the 
control  of  the  water  level  in  the  tank.  The  complexity  (requisite  variety)  that  this  creates  must  be  matched  by 
the  controller,  which  therefore  in  some  important  sense  must  mimic  the  essential  features  of  the  process. 

7.7.1.6  Requisite  Variety  as  Layers  of  Control 

As  a  generic  example,  consider  the  control  of  a  transportation  process,  such  as  driving  a  car  from  home  to 
work  or  guiding  an  unmanned  vehicle  to  a  destination.  In  both  cases  it  involves  the  movement  of  a  vehicle 
between  two  points,  A  and  B.  To  achieve  this  it  is  necessary  to  control  the  overall  progress  of  the  vehicle, 
i.e.,  to  ensure  that  it  gets  nearer  to  B  -  which  is  not  always  the  same  as  getting  further  from  A.  It  is  also 
necessary  to  ensure  that  the  vehicle  steers  clear  of  obstacles  in  the  close  environment,  i.e.,  that  it  does  not 
collide  with  any  stationary  or  moving  object,  and  that  it  negotiates  changes  in  speed  or  direction  in  an 
effective  manner.  It  is  furthermore  necessary  to  ensure  that  the  vehicle  has  the  necessary  resources  to  move, 
e.g.,  power.  It  is  finally  necessary  that  the  purpose  of  the  transportation  is  defined,  such  as  the  locations  of 
points  A  and  B,  the  criteria  for  effective  transportation  (e.g.,  speed,  power  consumption,  load,  etc.). 
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In  this  example  four  layers  of  control  were  required.  Other  models  have  often  suggested  three  different  types 
of  control,  such  as  strategy,  tactics,  and  operation  [15],  heading,  pursuit,  and  pre-cognitive  control  [16],  skill- 
based,  rule-based,  and  knowledge-based  performance  [17],  and  strategical  (planning),  tactical  (manoeuvring), 
and  operational  control  [18].  The  reason  for  proposing  four  layers  rather  than  three  is  that  for  most  activities 
the  layer  of  (operational)  control  presupposes  a  further  layer  where  actions  are  executed.  For  humans  this  is 
exemplified  by  what  we  do  automatically,  without  thinking  of  it  or  paying  attention  to  it.  Examples  are 
walking  or  running,  using  tools  (as  professionals  do),  maintaining  a  steady  speed  and  position  (in  driving),  etc. 
In  most  activities  execution  takes  place  so  quickly  that  we  are  not  aware  of  it  before  it  has  happened,  since 
attention  -  in  the  sense  of  the  act  of  becoming  aware  of  something  -  requires  a  certain  time. 

According  to  the  Law  of  Requisite  Variety,  the  controller  must  be  able  to  provide  the  several  types  of  control 
listed  above,  regardless  of  what  the  actual  number  is.  The  simplest  way  of  doing  that  is  obviously  to  assume 
that  the  functioning  of  the  controller  is  organised  in  different  layers  as  well.  In  this  manner  the  functional 
architecture  of  the  controller  embodies  part  of  the  variety  needed  to  control  the  process,  and  the  controlling 
system  therefore  to  some  extent  is  a  model  of  the  process.  This  obviously  needs  to  be  supplemented  by 
representations  of  the  specific  characteristics  of  the  domain,  the  context,  and  the  situation.  The  process  may 
also  require  that  there  multiple  goals  are  pursued  at  each  layer.  The  result  is  therefore  multiple,  simultaneous 
control  functions  organised  in  a  number  of  layers. 

7.7.2  Contextual  Control  Models  (COCOM) 

In  the  modelling  of  cognition,  human  or  otherwise,  a  distinction  can  be  made  between  procedural  prototype 
and  contextual  control  models.  A  procedural  prototype  model  assumes  that  a  pre-defined  sequence  of 
(elementary)  actions  or  a  procedural  pattern  exists,  and  that  this  represents  a  more  natural  way  of  doing  things 
than  others.  In  a  situation,  the  expected  next  action  can  therefore  be  found  by  referring  to  the  natural  ordering 
of  actions  implied  by  the  prototype.  Although  some  steps  may  be  bypassed  by  taking  shortcuts,  and  although 
the  procedural  pattern  may  be  applied  recursively,  the  underlying  sequence  itself  is  treated  as  immutable. 

A  contextual  control  model,  on  the  other  hand,  implies  that  actions  are  determined  by  the  context  rather  than 
by  an  inherent  sequential  relation  between  them.  In  a  situation,  the  next  action  is  therefore  determined  by  the 
current  context  and  by  the  competence  of  the  JCS  (cf.  below).  If  recurring  patterns  of  actions  are  found, 
this  can  be  ascribed  to  the  characteristics  of  the  environment  rather  than  pre-programmed  action  sequences. 

7.7.2.1  Dynamic  Control  -  COCOM 

Procedural  prototype  models  are  ubiquitous  in  human  factors  and  behavioural  sciences  and  examples  are  easy 
to  find.8  An  example  of  a  contextual  control  model  is  provided  by  the  COCOM,  which  describes  how  a 
controller  or  controlling  system  can  maintain  control  of  a  dynamic  process.9  In  this  model  the  basic  principle 
is  that  decisions  or  ‘actions’  are  determined  by  the  current  understanding  of  the  situation  (called  ‘construct’, 
cf.  Figure  7-15),  which  includes  the  anticipation  or  expectation  of  what  will  happen  next.  The  ‘events’ 
represent  the  result  of  the  actions  (hence  the  feedback).  If  they  match  the  expectations,  they  reinforce  the 
‘construct’;  if  there  is  a  mismatch,  the  ‘construct’  must  be  modified.  The  ‘events’  can  also  be  completely 
unexpected,  for  instance  if  they  are  due  to  disturbances;  that,  of  course,  also  demands  a  modification  of  the 


Typical  examples  are  models  of  decision  making  or  problem  solving. 

9  A  more  detailed  description  can  be  found  in  Hollnagel  [19]  and  Hollnagel  and  Woods  [12].  In  the  case  of  the  COCOM,  the  generic 
and  specific  names  are  unfortunately  identical. 
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‘construct’.  The  model  can  account  for  the  dynamics  of  a  situation,  specifically  the  consequences  of  lack  of 
time,  and  for  the  effects  of  different  levels  of  control. 
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Figure  7-15:  The  Basic  Construct-Action-Event  Cycle. 


1.1. 2.2  Extended  Control  Model  (ECOM) 

The  ECOM  describes  how  the  performance  of  a  joint  cognitive  system  (JCS)  takes  place  on  several  layers  of 
control  simultaneously,  using  the  notion  of  concurrent  control  loops.  The  ECOM  follows  the  same  principles 
of  modelling  as  the  COCOM  [19]  and  is  built  on  the  latter  in  the  sense  that  each  layer  corresponds  to  the 
fundamental  construct-action-event  cycle  depicted  in  Figure  7-15.  Some  of  these  are  closed  (reactive), 
some  are  open  (anticipatory),  and  some  are  mixed.  The  assumption  of  multiple  layers  of  activity  is  crucial  for 
the  modelling  approach,  and  it  has  in  practice  turned  out  that  four  layers  are  sufficient  [20],  although  there  is 
no  accepted  theory  that  determines  their  number.  Although  the  ECOM  has  been  developed  to  describe  joint 
cognitive  systems  that  include  human  operators,  it  has  also  been  used  to  describe  dynamic  systems  more 
generally,  both  technological  and  organisational.  In  the  following  sections,  each  of  the  four  layers  of  activity 
will  briefly  be  described,  going  from  the  lowest  to  the  highest  (cf.  Figure  7- 16). 10 

7. 7. 2. 2.1  Tracking 

The  tracking  layer  of  Figure  7-16  describes  the  activities  required  to  keep  a  JCS  inside  predetermined 
performance  boundaries,  typically  in  terms  of  safety  or  efficiency.  Activities  at  the  tracking  layer  are  very 
much  a  question  of  closed-loop  control.  For  the  skilled  user  such  activities  are  performed  automatically  and 
therefore  with  little  effort.  While  activities  at  the  tracking  layer  usually  are  performed  in  an  automatic  and 
unattended  manner,  they  may  become  attended,  hence  more  like  regulating,  if  conditions  change. 


10  A  more  extensive  treatment  can  be  found  in  Hollnagel  and  Woods  [12]. 
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In  the  ECOM,  goals  and  criteria  for  activities  at  the  tracking  layer  are  provided  by  the  regulating  layer. 
Most  of  the  tracking  activities  are  readily  amenable  to  technology  take-over  and  automation.  If  done  clumsily 
this  may  give  rise  to  automation  surprises  because  the  almost  complete  take-over  of  closed-loop  control  makes 
it  difficult  for  operators  to  follow  what  is  going  on,  hence  to  maintain  the  situation  comprehension  that  is 
needed  at  the  other  layers  of  activity. 

7. 7. 2. 2. 2  Regulating 

The  regulating  layer  describes  the  activities  by  which  a  JCS  achieves  short-term  goals,  such  as  specific 
manoeuvres  relative  to  the  environment  (which  need  not  be  physical  space).  Regulating  is  itself  basically  a 
closed-loop  activity,  although  anticipatory  control  may  also  occur  (e.g.,  [16]).  Activities  at  the  regulating  layer 
do  not  always  take  place  smoothly  and  automatically,  but  may  require  attention  and  effort.  These  activities  in 
turn  refer  to  specific  plans  and  objectives  that  come  from  the  monitoring  layer. 

Under  some  conditions,  the  regulating  loop  may  suspend  the  tracking  loop.  It  may,  for  instance,  be  more 
important  to  keep  maintain  integrity  than  to  follow  a  path.  This  can  also  be  expressed  as  a  temporary 
suspension  of  one  goal  (following  a  path)  to  the  advantage  of  another  (maintaining  integrity).  Incompatibility 
between  goals  can  be  resolved  by  changing  or  adjusting  plans. 

7. 7. 2. 2. 3  Monitoring 

Whereas  activities  at  the  regulating  layer  may  lead  to  either  direct  actions  or  goals  for  the  tracking  layer, 
activities  at  the  monitoring  layer  are  mainly  concerned  with  setting  objectives  and  activating  plans  for  actions. 
This  can  involve  monitoring  the  condition  of  the  vehicle,  although  this  has  in  most  cases  been  taken  over  by 
automation,  or  the  monitoring  the  state  of  the  environment. 

In  many  domains  a  distinction  between  regulating  (of  position)  and  monitoring  (of  location)  can  be  made, 
although  space  may  not  always  be  physical  (or  Euclidean).  In  a  vehicle,  other  activities  at  the  monitoring  layer 
may  be  related  to  information  sources.  Although  this  is  not  monitoring  of  performance  per  se ,  it  may  affect 
the  ability  to  perform,  particularly  if  it  is  non-trivial.  Monitoring  does  not  directly  influence  location  of  the 
vehicle  in  the  sense  of  closed-loop  control  and  regulation,  but  rather  is  concerned  with  the  state  of  a  JCS 
relative  to  its  environment. 

7. 7. 2. 2. 4  Targeting 

The  last  type  of  action  occurs  at  the  targeting  or  goal  setting  layer.  An  obvious  kind  of  goal-setting  is  with 
regard  to  the  destination.  That  goal  may  give  rise  to  several  subgoals  and  activities,  some  of  which  can  be 
automated  or  supported.  Other  goals  have  to  do  with  criteria  for  acceptable  performance. 

Goal-setting  is  distinctly  an  open-loop  activity,  and  is  implemented  by  a  nontrivial  set  of  actions  that  often 
covers  an  extended  period  of  time.  Assessing  the  change  relative  to  the  goal  is  not  based  on  simple  feedback, 
but  rather  on  a  loose  assessment  of  the  situation  -  for  instance,  proximity  to  target.  When  the  assessment  is 
done  regularly  it  may  be  considered  as  being  a  part  of  monitoring  and  control.  If  the  assessment  is  done 
irregularly,  the  trigger  is  usually  some  unknown  factor,  perhaps  time,  perhaps  a  pre-defined  cue  or  landmark 
(physical  or  symbolic),  perhaps  a  background  ‘simulation’  or  estimation  of  the  general  progress  (like  suddenly 
feeling  uneasy). 

Figure  7-16  shows  the  relations  among  the  four  layers  in  a  simplified  manner.  To  avoid  graphical  clutter, 
Figure  7-16  includes  only  the  goal  dependencies  among  the  layers;  other  dependencies  exist,  for  instance, 
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in  the  propagation  of  feedback  or  events.  For  the  same  reason  each  layer  is  represented  only  by  one  construct- 
action-event  cycle,  even  though  there  normally  will  be  several  concurrent  cycles  or  loops.  The  arrows  at  the 
right-hand  side  of  Figure  7-16  indicate  the  relative  weight  of  feedback  and  feedforward  control  for  each  layer. 
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Figure  7-16:  The  Extended  Control  Model  (ECOM). 


1.1.23  ECOM  Structure  and  Parameters 

The  main  characteristics  of  each  of  the  four  layers  are  summarised  in  Table  7-7.  This  shows  for  each  layer  the 
type  of  control  involved,  the  typical  demands  to  attention  in  the  case  of  a  human  controller,  how  often  events 
can  be  expected  to  occur,  and  finally  the  typical  duration  of  events.  Here  it  is  important  to  note  that  events  on 
the  tracking  layer  usually  are  of  such  short  duration  that  in  the  case  of  humans  they  are  pre-attentive.  In  other 
words,  tracking-type  behaviour  is  equivalent  to  skills,  in  the  sense  that  it  is  something  that  is  done  more  or  less 
automatically  and  without  attention.  The  regulating  layer  comprises  actions  of  a  short  duration  that  do  require 
attention,  but  not  for  long.  The  monitoring  layer  describes  actions  that  go  on  intermittently  as  long  as  the  task 
lasts,  although  the  distribution  can  be  decidedly  irregular,  depending  on  demands.  Finally,  actions  on  the 
targeting  layer  take  place  every  now  and  then,  almost  always  including  the  preparation  of  a  task. 
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Table  7-7:  Functional  Characteristics  of  ECOM  Layers 


Control  Layer 

Tracking 

Regulating 

Monitoring 

Targeting 

Type  of  control 
involved 

Feedback 

Feedforward 
and  feedback 

Feedback 

(condition 

monitoring) 

Feedforward 
(goal  setting) 

Demands  to 
attention 

None 

(pre-attentive) 

High  (uncommon 
actions);  low 
(common  actions) 

Low,  intermittent 

High, 

concentrated 

Frequency 

Continuous 

Medium  to  high 

(context 

dependent) 

Intermittent, 
but  regular 

Low 

(preparations, 

re-targeting) 

Typical  duration 

<  1  sec 

(‘instantaneous’) 

1  sec  -  1  minute 
(‘short  term’) 

10  minutes  - 
task  duration 
(‘long  term’) 

Short  (minutes) 

The  ECOM  can  be  used  to  describe  the  interactions  between  the  different  layers.  The  assumption  throughout 
is  that  all  layers  are  active  simultaneously,  or  rather  that  goals  and  objectives  corresponding  to  different  layers 
of  control  are  being  pursued  simultaneously.  One  use  of  the  ECOM  is  therefore  to  account  for  the  nontrivial 
dependence  between  goals  and  activities  among  the  layers,  for  instance  when  the  tracking  layer  is  interrupted 
by  an  unexpected  disturbance.  The  goals  of  each  control  loop  can  also  be  temporarily  suspended  as  when  a 
higher-level  goal  is  suspended  in  lieu  of  focusing  on  a  lower  level  one.  The  bottom  line  is  that  controller 
performance  and  the  effect  of  partial  autonomy  can  be  understood  only  in  the  context  of  the  JCS  as  a  whole, 
and  not  at  the  level  of  individual  components  or  parts. 

Returning  to  the  issue  of  automation,  it  is  clearly  possible  to  consider  the  possibility  of  automation  at  each 
layer  of  control.  In  some  cases  functions  have  to  be  automated  because  it  is  impossible  for  humans  to  carry 
them  out  with  sufficient  speed,  precision,  or  stability.  In  other  cases  functions  have  to  be  carried  out  by 
humans  because  the  environment  is  too  uncertain.  In  yet  other  cases  there  may  be  a  choice.  The  top-down 
approach  represented  by  the  layers  of  control  differs  from  the  traditional  way  of  comparing  humans  and 
machines  because  the  comparison  refers  to  the  demands  of  the  tasks  rather  than  to  the  capabilities  of  system 
components.  Table  7-7  can  therefore  also  be  used  as  a  way  of  considering  automation.  In  order  to  do  this  it  is, 
however,  necessary  to  enter  into  some  other  considerations. 

7.7.3  Autonomous  Control  Level  Framework 

The  concept  of  Autonomous  Control  Levels  (ACL)  has  been  developed  to  characterise  issues  of  autonomous 
control  in  UAV  missions.  According  to  Huang  et  al.  [21]  the  three  main  motivations  for  higher  autonomy  in 
unmanned  system  are: 

1)  Lack  of  bandwidth  required  for  extensive  tele-operation; 

2)  Safety  for  personnel;  and 

3)  That  mission  effectiveness  may  be  limited  by  cognitive  workload,  i.e.,  that  humans  are  limited  in  their 
ability  to  perform  their  own  tasks  within  the  mission. 
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To  that  might  be  added  time  delays  associated  with  communication  links,  which  may  hamper  feedback 
control.  A  further  motivation  is  that  unmanned  systems  currently  require  control  by  one  or  more  highly  trained 
human  operators;  increasing  the  level  of  autonomy  would  clearly  reduce  this  demand.  All  in  all,  increased 
autonomous  control  is  seen  as  a  way  of  maximising  UAV  utility.11 

7.7.3. 1  Levels  of  Autonomy 

The  ACL  framework  comprises  ten  autonomous  control  levels  are  shown  in  Figure  7-17.  While  the  labels  for 
the  different  levels  are  evocative,  it  is  difficult  to  find  clear  definitions  of  the  terms,  even  in  the  Roadmap 
report  [22].  In  a  discussion  of  the  ACL,  Reising  [23]  pointed  out  that  the  levels  seemed  to  be  based  on  a 
conglomeration  of  levels  of  automation,  human  information  processing  models,  and  adaptive  automation. 
While  attempts  have  been  made  to  explain  the  ACL  in  terms  of  concepts  such  as  automation,  workload, 
situation  awareness  (e.g.,  [24],  but  see  also  [25]),  Reising  (op.  cit.)  pointed  out  that  this,  as  well  as  the  original 
proposal  for  the  ten  levels  of  automation  [5],  focused  on  the  interaction  between  the  operator  and  the  UAV 
rather  than  on  the  autonomy  of  a  vehicle  or  a  JCS  considered  as  a  whole.  The  work  of  Kaber  and  Endsley  [24] 
furthermore  started  from  the  levels  of  automation,  hence  did  not  provide  an  explanation  of  the  nature  of  the 
levels  nor  their  number.  Knowing  the  development  of  descriptions  of  automation  and  autonomy  it  is,  however, 
safe  to  assume  that  the  reason  for  having  ten  levels  can  be  found  in  1978  proposal,  which  quickly  achieved  an 
almost  mythical  status  -  even  though  their  number  was  later  reduced  to  eight  [6]. 


Autonomous  Control  Levels 


11  In  practice,  the  issue  is  one  of  automation  rather  than  autonomy.  An  autonomous  system  is,  by  definition,  completely 
independent,  hence  beyond  human  control.  That  is  neither  possible,  nor  desirable. 
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Figure  7-17  shows  the  ten  levels  of  autonomy  as  developing  gradually,  beginning  with  remotely  guided 
vehicles  in  the  late  1950s  and  ending  with  fully  autonomous  swarms  of  vehicles  in  2025. 12  Superimposed  on 
this  development  is  a  characterisation  of  three  mission  types  or  configurations,  namely  individual  (single 
vehicle),  group  (multiple  vehicles),  and  team  or  swarm  (multiple  vehicles  performing  as  a  single  entity). 
An  important,  but  unspoken,  assumption  is  that  utility  increases  proportionally  with  the  level  of  autonomy, 
hence  that  a  high  level  of  autonomy  is  more  desirable  than  a  low  level.  The  ACL  framework  also  implies 
bottom-up  inheritance,  i.e.,  that  capabilities  that  exist  at  lower  levels  of  autonomy  are  carried  forth  to  higher 
levels. 

1.13.2  Individuals,  Groups,  Swarms 

In  Figure  7-17,  the  three  mission  types  -  individual,  group,  and  team  (swarm)  -  are  mapped  as  corresponding 
directly  to  increasing  levels  of  autonomy.  On  further  reflection,  however,  this  need  not  be  so. 

As  far  as  the  three  mission  types  are  concerned,  the  meaning  of  an  individual  vehicle  is  obvious,  i.e.,  it  is  a 
mission  carried  out  by  a  single  vehicle.  The  difference  between  group  and  team ,  where  the  latter  is  taken  to  be 
synonymous  with  swarm ,  is  less  obvious.13  As  noted  by  Reising  [23],  a  swarm  does  not  appear  to  comprise  a 
central  controller  that  tells  swarm  what  to  do.  As  seen  in  the  natural  world  -  schools  of  fish,  flocks  of  birds, 
swarms  of  bees  -  “...  swarming  itself  is  a  type  of  emergent  behavior,  a  behavior  that  isn’t  explicitly 
programmed,  but  results  as  a  natural  interaction  of  multiple  entities”  [26,  p.  1].  A  swarm  consists  of  a  large 
number  of  members  that  all  behave  in  a  similar  way,  i.e.,  which  have  the  same  functions  and  capabilities. 
In  particular,  the  members  are  replaceable. 

On  the  other  hand,  a  group  is  defined  in  social  psychology  as  an  assembly  of  individuals,  recognised  as  being 
individually  different,  but  treated,  for  some  purpose,  as  parts  of  a  larger  unit.  Group  members  usually  have 
characteristic  individual  behaviours  that  in  combination  enable  the  group  to  achieve  its  purpose.  The  principal 
difference  between  a  group  and  a  team/swarm  is  consequently  that  group  members  can  exhibit  different 
(specialised)  behaviours,  while  all  members  of  a  swarm  behave  in  essentially  the  same  way.  In  a  group, 
the  behaviour  of  the  members  must  therefore  be  explicitly  coordinated;  in  a  swarm,  the  members  behave 
collectively.  Although  the  4 intelligence’  of  a  swarm  may  be  ‘emergent’,  it  does  not  necessarily  mean  that  it  is 
more  difficult  to  control. 


1.133  Mission  Type  and  Autonomy 

In  the  original  for  layers  of  automation  [5],  the  end  points  of  the  scale  were  complete  manual  control  and 
complete  automation,  respectively.  In  relation  to  the  mission  type,  it  is  obvious  that  an  individual  vehicle  can 
range  from  one  extreme  to  the  other,  specifically  that  it  in  principle  can  be  completely  automated  -  but  not 
completely  autonomous.  The  same  goes  for  swarms,  although  the  manual  control  must  address  the  swarm  as  a 
unit  rather  than  the  single  entities  that  make  up  the  swarm.  In  the  case  of  a  group,  it  is  probably  not  feasible  to 
consider  manual  control,  since  the  coordination  demands  will  be  considerable  and  in  a  sense  lead  to  the 
deconstruction  of  the  group.  On  the  other  hand,  there  is  nothing  that,  in  principle,  makes  complete  automation 
impossible.  (A  group  might,  in  fact,  be  the  only  unit  that  can  achieve  autonomy.) 


12 

The  development  is  also  shown  as  going  faster  and  faster.  This  may  be  wishful  thinking  rather  than  reality. 

13 

Note,  however,  that  the  major  groupings  have  no  clear  correspondence  with  the  categories  on  the  Y-axis.  Levels  5-9  refer  to  the 
group,  while  only  level  1 0  refers  to  the  swarm. 
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These  considerations  suggest  that  the  ACL  should  be  considered  in  terms  of  the  two  dimensions  of  mission 
type  and  level  of  automation,  rather  than  in  terms  of  level  of  autonomy  alone.  The  result  of  this  exercise  is 
shown  in  Table  7-8,  where  the  level  of  automation  simply  is  an  ordinal  number,  with  no  inherent  meaning. 


Table  7-8:  A  Possible  Two-Dimensional  Description  of  Autonomous  Control  Levels 


Level  of 

Mission  Type 

Automation 

Individual 

Group 

Team/Swarm 

7 

Strategic  goals  and 
replan 

Strategic  goals  and 
replan 

Strategic  goals  and 
replan 

6 

Tactical  goals  and 
replan 

Tactical  goals  and 
replan 

Tactical  goals  and 
replan 

5 

Onboard  route  replan 
(adapt  to  route 
changes) 

Onboard  route  replan 
(adapt  to  route 
changes) 

Onboard  route  replan 
(adapt  to  route 
changes) 

4 

Coordinated  control 

Collective  control 

3 

Adapt  to  failure  and 
flight  conditions 
(“see-and-avoid”) 

Adapt  to  failure  and 
flight  conditions 
(“see-and-avoid”) 

Adapt  to  failure  and 
flight  conditions 
(“see-and-avoid”) 

2 

Real-time  health  / 
diagnosis 

Real-time  health  / 
diagnosis 

Real-time  health  / 
diagnosis 

1 

Manual  control 
(remotely  guided) 

Manual  control 
(remotely  guided) 

Although  the  distinction  between  strategic  and  tactical  is  firmly  entrenched  in  military  language,  it  is  from  a 
scientific  point  relative  rather  than  absolute.  The  issue  is  basically  one  of  defining  the  goals  for  the  system 
under  consideration,  whether  it  is  an  individual  UAV,  a  group  or  a  team.  If  a  target  is  set  with  a  long  span-of- 
foresight  it  corresponds  to  a  strategy  [15];  if,  on  the  other  hand  a  target  is  set  with  a  short  span-of-foresight 
and/or  change  frequently,  it  corresponds  to  a  tactic.  (In  Shiitzenberger’s  terms,  “the  optimal  strategy  is  just  the 
simple  tactic  of  attempting  to  do  one’s  best  on  a  purely  local  basis”.)  The  shortest  span-of-foresight  is  in  route 
revision,  which  therefore  can  be  seen  as  a  low-level  tactic.  For  group  missions,  route  revision  requires 
coordination;  for  team/swarm  missions,  route  revision  can  be  achieved  by  collective  control. 

The  ECOM  described  four  layers  of  control  called  targeting,  monitoring,  regulating  and  tracking,  respectively. 
If  we  consider  the  modified  description  of  the  ACL  categories  shown  in  Table  7-8,  it  is  possible  to  propose  a 
mapping  between  the  ACL  and  the  ECOM,  as  shown  in  Table  7-9.  Here  the  two  highest  levels  of  autonomy  - 
or  rather,  automation  -  are  seen  as  corresponding  to  the  targeting  layer  in  the  ECOM.  The  issue  is  one  of 
defining  the  goals  for  the  system  under  consideration,  whether  it  is  an  individual  UAV,  a  group  or  a  team. 
It  may  well  be  that  targeting  capabilities  will  not  be  considered  for  individual  missions. 
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Table  7-9:  Relations  between  ECOM  and  the  Revised  ACL  Description 


ECOM  Layer 

Mission  Type 

Individual 

Group 

Team/Swarm 

Targeting 

Goal  setting  and 
replan  (strategic, 
tactical) 

Goal  setting  and 
replan  (strategic, 
tactical) 

Goal  setting  and 
replan  (strategic, 
tactical) 

Onboard  route  replan 
(adapt  to  route 
changes) 

Onboard  route  replan 
(adapt  to  route 
changes) 

Onboard  route  replan 
(adapt  to  route 
changes) 

Monitoring 

Adapt  to  failure  and 
flight  conditions 
(“see-and-avoid”) 

Adapt  to  failure  and 
flight  conditions 
(“see-and-avoid”) 

Adapt  to  failure  and 
flight  conditions 
(“see-and-avoid”) 

Real-time  health  / 
diagnosis 

Real-time  health  / 
diagnosis 

Real-time  health  / 
diagnosis 

Coordinated  control 

Collective  control 

Regulating 

Manual  control 
(remotely  guided) 

Manual  control 
(remotely  guided) 

Tracking 

The  second  part  of  the  targeting  layer  is  on-board  route  revision,  i.e.,  the  ability  to  generate  or  select  an 
alternative  route.  The  impetus  to  do  that  can  come  either  from  the  monitoring  of  flight  conditions  or  from  an 
external  source  of  command  (a  revision  of  strategic  or  tactical  goals).  Route  revision  is  clearly  relevant  for 
single  UAVs  as  well  as  for  multiple  UAVs.  As  regards  multiple  UAVs,  the  difference  may  be  whether  they 
are  individually  controlled  (as  in  a  group),  or  whether  they  are  seen  as  a  larger  unit  (e.g.,  a  swarm),  where 
individual  behaviour  is  subsumed  the  behaviour  of  the  swarm.  In  other  words,  in  a  group  single  UAVs  may 
have  different  roles  and  responsibilities,  whereas  a  swarm  functions  as  a  collective  whole.)  On  the  group  level 
this  corresponds  to  coordination  and  distributed  control,  on  the  team  level  to  collective  control. 

The  next  ACL  levels,  real-time  health/diagnosis  and  adapting  to  failures  and  flight  conditions  (“see-and- 
avoid”)  correspond  to  the  monitoring  layer  of  the  ECOM.  Monitoring  is  clearly  relevant  for  all  three  mission 
types.  Indeed,  autonomous  monitoring  is  imperative  for  missions  with  multiple  vehicles,  since  human 
monitoring  of  simultaneous  activities  is  not  reliable.  Monitoring  can  be  either  of  the  vehicle  itself  or  of  the 
vehicle’s  environment  (threats). 

As  argued  above,  regulating  -  in  the  sense  of  manual  control  -  is  relevant  mostly  for  the  individual  mission 
type.  To  do  the  same  for  a  group  of  vehicles  will  require  extensive  coordination  and  therefore  not  be  feasible. 
It  may  still  be  possible  for  a  swarm,  since  that  can  be  considered  as  one  unit  to  be  regulated.  Even  so, 
the  efforts  involved  will  not  be  the  same  as  for  an  individual  vehicle. 

In  Table  7-9,  regulation  occurs  in  two  different  meanings.  In  case  of  an  individual  vehicle,  regulation 
corresponds  to  conventional  remote  guidance  or  manual  control.  In  the  case  of  groups  and  teams/swarms  low 
level  regulation  of  this  type  is  not  desirable.  On  the  contrary,  it  is  probably  a  precondition  for  group  and  team 
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missions  to  be  possible  that  the  control  of  basic  manoeuvres  is  completely  automated.  There  is,  however, 
another  level  of  regulation,  which  has  to  do  with  the  proper  coordination  of  group  or  team  members. 
This  coordination  is  required  in  order  to  be  able  to  implement  route  changes  quickly  and  efficiently,  i.e.,  a  sort 
of  guidance  at  a  higher  level.  In  the  case  of  groups  and  teams,  both  monitoring  and  regulating  could  be 
extensively  automated,  leaving  targeting  in  the  hands  of  human  operators. 

Finally  the  layer  of  tracking  is  assumed  to  be  completely  automated,  as  it  already  is  today.  For  airborne 
vehicles  tracking  (roll,  pitch,  yaw)  must  happen  so  quickly  that  it  exceeds  human  capability.  This  layer  will 
therefore  not  be  considered  further  here. 

One  consequence  of  the  above  considerations  is  that  Figure  7-17  should  be  revised  to  show  the  three  mission 
types  in  parallel  relative  to  the  control  levels  rather  than  in  series.14  In  other  words,  they  represent  three  lines 
of  development  rather  than  one.  There  is,  indeed,  no  good  reason  why  a  team  should  be  seen  as  more  difficult 
to  automate  than  a  group.  In  fact,  it  may  well  be  the  reverse,  since  a  group  requires  the  coordination  of  several 
individual  behaviours  rather  than  of  a  “single”  behaviour  of  either  an  individual  or  a  team/swarm  seen  as  a 
“collective  individual.”15  A  suggestion  of  what  this  revision  might  look  like  is  shown  in  Figure  7-18,  which 
also  reverses  the  ACL  ordering  of  team  and  group. 


Autonomous  Control  Levels 


Figure  7-18:  Revised  ACL  Description. 


On  the  other  hand,  this  means  that  they  will  appear  sequentially  with  respect  to  time,  rather  than  as  being  developed  in  parallel. 
That  may  be  a  not  entirely  unwanted  side- effect. 

15  All  this  assumes,  of  course,  that  the  intentions  of  the  roadmap  have  been  correctly  interpreted. 
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7.7.4  Analysis  of  UAV  Scenarios 

The  use  of  the  concept  of  layers  of  control  will  be  illustrated  by  describing  an  UAV  scenario.  The  description 
will  address  three  characteristic  levels  of  autonomy  defined  by  the  ‘Roadmap’,  namely  ACL1,  ACL6,  and 
ACL9.  In  terms  of  the  descriptions  given  above,  this  corresponds  to  a  remotely  guided,  single  vehicle  mission; 
a  group  mission;  and  a  team  mission. 

Table  7-10  shows  the  scenario  that  is  used.  It  divides  the  mission  into  the  three  phases  of  ‘ingress’,  ‘over 
target’,  and  ‘egress’,  where  each  phase  is  described  further  in  terms  of  a  number  of  tasks  or  functions. 


Table  7-10:  Overall  Description  of  UAV  Scenario 


INGRESS 

Control,  Guidance,  Navigation  (own  ship  and  attack  aircraft) 

Replan 

Communication  (C2  MC,  attack  aircraft,  other  UAV  UCS) 

System  management  (+contingencies) 

Self  defence 

Target  location 

OVER 

TARGET 

Target  registration,  identification,  verification,  designation 
(for  attack  aircraft) 

Control,  Guidance,  Navigation 

System  management  (+contingencies) 

Communication  (C2  MC,  attack  aircraft,  other  UAV  UCS) 

Sensor  management 

Self  defence 

Rules  of  engagement 

Battle  damage  assessment 

EGRESS 

Control,  Guidance,  Navigation 

Communication  (C2  MC,  attack  aircraft,  other  UAV  UCS) 

System  management  (+contingencies) 

Self  defence 

One  thing  to  notice  about  this  scenario  description  is  that  a  number  of  tasks  are  common  to  all  three  phases. 
These  are:  (1)  Control,  Guidance,  Navigation  (own  ship  and  attack  aircraft);  (2)  Communication  (C2  MC, 
attack  aircraft,  other  UAV  UCS);  (3)  System  management  (+contingencies);  and  (4)  Self  defence. 
It  is  therefore  reasonable  to  describe  the  scenario  in  terms  of  specific  and  common  tasks,  cf.  Table  7-11. 
This  shows  that  only  the  ‘ingress’  and  ‘over  target’  phases  comprise  specific  tasks.  It  would  be  possible  to 
analyse  the  scenario  in  more  detail  using  a  recognised  task  analysis  method,  such  as  hierarchical  task  analysis 
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or  goals-means  task  analysis.  This  would  actually  be  necessary  to  perform  a  complete  analysis  of  the 
command  and  control  demands  of  the  scenario,  but  it  cannot  be  done  on  the  basis  of  the  available  material. 
Further  details,  as  well  as  expertise  knowledge,  would  be  required.  For  the  present  purpose  the  available 
material  is,  however,  sufficient  to  illustrate  how  the  tasks  that  make  up  the  scenario  can  be  characterised  in 
terms  of  different  control  layers,  as  a  basis  for  anticipating  the  effects  of  increased  automation  (autonomy). 


Table  7-11:  Common  and  Specific  Tasks  in  the  UAV  Scenario 


SPECIFIC  TASKS 

COMMON  TASKS 

INGRESS 

Replan 

Control,  Guidance,  Navigation 
(own  ship  and  attack  aircraft) 

Target  location 

Communication  (C2  MC,  attack 
aircraft,  other  UAV  UCS) 

System  management 
(+contingencies) 

Self  defence 

OVER 

TARGET 

Target  registration,  identification, 
verification,  designation 
(for  attack  aircraft) 

Control,  Guidance,  Navigation 
(own  ship  and  attack  aircraft) 

Sensor  management 

System  management 
(+contingencies) 

Rules  of  engagement 

Communication  (C2  MC,  attack 
aircraft,  other  UAV  UCS) 

Battle  damage  assessment 

Self  defence 

EGRESS 

Control,  Guidance,  Navigation 
(own  ship  and  attack  aircraft) 

Communication  (C2  MC,  attack 
aircraft,  other  UAV  UCS) 

System  management 
(+contingencies) 

Self  defence 

7.7.4.1  Description  of  Common  Tasks 

The  four  common  tasks  of  this  scenario  are: 

•  Control,  guidance,  and  navigation  (own  ship  and  attack  aircraft).  This  is  a  composite  task,  which  has 
at  least  the  three  components  listed  here.  We  shall  further  focus  on  the  vehicle  itself,  since, 
e.g.,  guidance  of  attack  aircraft  reasonably  might  be  subsumed  under  communication.  In  this  case 
‘control’  is  understood  as  corresponding  to  tracking,  i.e.,  keeping  yaw,  pitch,  roll,  etc.,  within  design 
and  operational  limits.  As  such  it  is  clearly  something  that  is  done  automatically  by  the  vehicle,  as  it 
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is  for  all  modern  aircraft.16  Guidance  and  navigation  involve  the  layers  of  monitoring  and  regulation. 
The  control  inputs  to  the  vehicle  will  typically  be  in  terms  of  the  course,  speed,  and  altitude  needed  to 
execute  specific  manoeuvres  or  reach  specific  goals.  The  goals  themselves  are  both  those  that  are 
defined  by  the  mission  (pre-planned)  and  those  that  arise  as  a  result  of  contingencies,  i.e.,  system 
management  and  self  defence. 

•  Communication  (C2  MC,  attack  aircraft,  other  UAV  UCS).  This  is  understood  to  comprise 
maintaining  contact  with  the  base  as  well  as  communicating  with  other  entities  involved  in  the 
mission  (attack  aircraft,  etc.).  It  can  therefore  be  seen  as  part  of  the  monitoring  -  and  to  some  extent 
of  the  regulating  -  of  the  mission  as  a  whole,  rather  than  tasks  that  specifically  are  part  of  what  the 
vehicle  does.  The  mission  of  the  vehicle  may  indeed  be  seen  as  serving  the  purpose  of 
communication,  i.e.,  providing  information  as  a  necessary  input  to  a  superordinate  task,  i.e.,  that  of 
achieving  the  larger  mission  objective. 

•  System  management  (+contingencies).  These  are  clearly  tasks  that  correspond  to  the  monitoring  layer 
of  the  ECOM.  As  mentioned  above,  monitoring  can  be  of  the  vehicle  itself  as  well  as  of  the 
environment.  In  that  later  case  it  may  lead  to  the  formation  of  new  goals,  cf.  below. 

•  Self  defence.  In  order  for  the  vehicle’s  mission  to  be  accomplished,  it  must  obviously  be  able  to 
maintain  its  functional  integrity,  which  means  the  ability  to  defend  itself  in  various  ways.  The  defence 
can,  for  instance,  be  by  evasive  manoeuvres  or  by  direct  counteraction.  In  relation  to  the  ECOM, 
self  defence  can  be  seen  as  a  targeting  activity,  i.e.,  the  formation  of  situation  specific  goals  -  which 
may  possibly  supersede  pre-defined  mission  goals.  Needless  to  say,  targeting  is  based  on  processing 
of  incoming  information,  hence  monitoring.  A  new  goal  established  in  this  way  will  give  rise  to  new 
loops  on  the  monitoring  and  regulating  layers,  cf.  the  description  above. 

The  relations  among  the  ECOM  layers  and  the  common  mission  tasks  is  summarised  in  Table  7-12. 

Table  7-12:  ECOM  Characterisation  of  Common  Mission  Tasks 


Common  Mission  Tasks 

Corresponding  ECOM  Layer 

Targeting 

Monitoring 

Regulating 

Tracking 

Control,  guidance,  navigation 

X 

X 

Communication  (C2  MC,  attack  aircraft, 
other  UAV  UCS) 

X 

(X) 

Autonomous 

System  management  (+  contingencies) 

X 

Self  defence 

X 

(X) 

7.7  .4 .2  Description  of  Specific  Tasks 

The  six  specific  tasks  are  found  in  the  ‘ingress’  and  ‘over  target’  phases. 

•  Ingress:  Replan.  In  the  original  task  description  this  appears  as  a  subsidiary  task  to  ‘control,  guidance, 
navigation’.  It  is  therefore  appropriate  to  see  it  as  associated  with  targeting,  or  rather  re-targeting. 

16  Similar  developments  are  found  in  land-based  vehicles,  even  for  civilian  purposes,  and  in  maritime  vehicles. 
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It  takes  place  in  response  to  an  external  command,  and  as  a  result  leads  to  changes  (i.e.,  new  goals) 
for  both  monitoring  and  regulating. 

•  Ingress:  Target  location.  This  is  a  specific,  composite  activity  which  best  can  be  interpreted  as  higher- 
level  regulating.  It  is  composite  in  the  sense  that  it  comprises  monitoring  of  the  current  position  and 
of  the  environment,  in  order  to  recognise  the  target.  This  may  in  turn  require  specific  manoeuvres 
(regulating)  to  bring  the  vehicle  sufficiently  close  to  the  target.  Target  location  thus  entails  both 
monitoring  and  regulating. 

•  Over  target:  Target  registration,  identification,  verification,  designation  (for  attack  aircraft).  One  of 
the  main  goals  of  an  UAV  is  to  identify  a  target  and  to  communicate  that  information  to  the  mission 
centre  or  to,  e.g.,  an  attack  aircraft.  In  terms  of  the  ECOM,  target  registration  is  mainly  a  question  of 
monitoring  the  environment  in  combination  with  communication  (as  a  common  mission  task). 

•  Over  target:  Sensor  management.  This  is  understood  as  being  the  management  of  the  various  types  of 
information  (sensors,  channels),  to  make  the  best  use  of  the  available  information  for  the  current 
goal/action.  It  can  thus  be  seen  as  a  kind  of  internal  regulating,  not  of  the  vehicle’s  behaviour, 
but  rather  the  selection  and  processing  of  sensor  input  to  be  optimally  responsive  to  the  needs  of  the 
current  task.  In  terms  of  the  ECOM,  sensor  management  is  a  special  kind  of  regulating. 

•  Over  target:  Rules  of  engagement.  The  rules  of  engagement  (ROE)  provide  guidance  governing  the 
use  of  force  consistent  with  mission  accomplishment  [27].  The  ROE  can  therefore  be  seen  as  defining 
part  of  the  performance  envelope  for  the  vehicle,  specifically  the  behaviour  that  under  normal 
circumstances  is  prohibited.  In  order  to  meet  the  ROE  it  is  therefore  necessary  to  monitor  the  situation 
and  to  abandon  specific  actions  if  they  are  in  conflict  with  the  ROE.  Seen  as  a  task,  ‘rules  of 
engagement’  therefore  entail  both  monitoring  and  regulating. 

•  Over  target:  Battle  damage  assessment.  This  is  a  straightforward  case  of  monitoring  for  specific 
changes.  It  may  produce  specific  manoeuvres  as  a  way  of  getting  the  recording  the  needed  data.  Battle 
damage  assessment  thus  comprises  monitoring  and  possibly  regulating. 

The  relations  among  the  ECOM  layers  and  the  specific  mission  tasks  is  summarised  in  Table  7-13. 


Table  7-13:  ECOM  Characterisation  of  Specific  Mission  Tasks 


Specific  Mission  Tasks 


Corresponding  ECOM  Layer 


Targeting 

Monitoring 

Regulating 

Tracking 

Replan 

X 

Target  location 

X 

X 

Target  registration,  identification, 
verification,  designation 
(for  attack  aircraft) 

X 

Sensor  management 

X 

(internal) 

Rules  of  engagement 

X 

X 

Battle  damage  assessment 

X 

(X) 
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7.7.5  Control,  Automation  and  Views 

Designing  for  control  can  be  viewed  in  different  ways,  ranging  from  graphical  interface  design  to  intelligent, 
autonomous  agents  or  even  virtual  reality  avatars.  Within  cognitive  systems  engineering,  understanding  the 
nature  of  control  is  a  prerequisite  for  the  design  of  the  elements  of  control  such  as  interfaces  and  information 
presentation.  As  mentioned  above,  the  essence  of  control  is  that  unwanted  variability  in  the  process,  which 
may  lead  to  deviations  from  the  desired  or  prescribed  course  of  development,  is  reduced  to  a  minimum  or  does 
not  occur  at  all.  Effective  control  therefore  requires  the  ability  to  detect  such  variability  and  to  respond  to  it, 
which  in  most  cases  also  means  the  ability  to  anticipate  large  fluctuations  and  prevent  them  from  happening. 
A  system  for  effective  control  must  therefore  support  three  different  types  of  view:  a  view  of  what  has 
happened  (the  past),  a  view  of  what  happens  here  and  now  (the  present),  and  a  view  of  what  may  possibly 
happen  (the  future). 

If  the  notion  of  the  three  different  views  is  applied  to  the  four  layers  of  control,  it  is  evident  that  not  all  views 
are  important  at  each  layer.  Targeting,  for  instance,  is  concerned  with  developing  plans  for  future  actions, 
while  tracking  is  more  about  responding  to  changes  in  the  situation.  Indeed,  the  differences  in  the  type  of 
control  (feedback,  feedforward)  involved  at  each  layer  give  rise  to  different  demands  for  data  and  information. 
A  proposal  for  the  relative  importance  of  the  three  different  views  is  shown  in  Table  7-14. 


Table  7-14:  Relations  between  Control  Layers  and  Process  Views 


Control  Layers 

View 

Tracking 

(feedback  control) 

Regulating  (feedforward 
+  feedback  control) 

Monitoring 
(feedback  control) 

Targeting 

(feedforward 

control) 

Past 

Important 

Very  important 

Present 

Very  important 

Very  important 
(Feedback) 

Very  important 

Future 

Very  important 
(Feedforward) 

Very  important 

Getting  the  information  provided  by  each  view  involves  a  cost,  most  obviously  in  terms  of  time.  Making  use 
of  the  information  -  processing  it,  understanding  it  -  also  takes  time,  which  generally  increases  with  the 
complexity  of  the  information.  Since  the  process  that  is  controlled  usually  cannot  wait,  the  management  of 
time  is  a  critical  issue  in  the  design  of  control  systems  [28,12].  This  is  certainly  the  case  for  the  control  of  a 
vehicle,  not  least  if  it  is  uninhabited. 

7.7.5.1  Control  and  Time 

At  the  tracking  and  regulating  layers,  time  is  very  short  -  and  may  in  some  cases  be  so  short  that  humans 
hardly  can  be  involved.  A  driver  can  steer  a  car  (tracking  layer  control)  when  (s)he  is  in  it,  mainly  because 
(s)he  is  able  directly  to  perceive  the  environment  -  at  least  during  daylight.  It  would  be  very  hard  to  do  this  by 
remote  control,  since  important  cues  such  as  motion  feedback  are  missing,  even  in  a  full-blown  synthetic 
sensory  environment.  At  the  regulating  layer  human  control  may  be  possible  depending  on  the  speed  of  the 
vehicle  (a  ship,  a  land  vehicle,  a  flying  drone),  although  in  most  cases  it  is  still  impractical.  At  both  layers 


7-92 


RTO-TR-HFM-078 


dMNATO 
WP  I  OTAN 


HUMAN  AUTOMATION  INTEGRATION 


automation  may  therefore  be  necessary  to  achieve  sufficiently  fast  performance;  modern  cars,  especially  at  the 
high-end,  are  a  good  example  of  that.  At  the  monitoring  and  targeting  layers  the  demands  to  rapid  responses 
are  smaller,  but  automation  may  be  needed  for  other  reasons,  e.g.,  to  achieve  steadiness  and  regularity  of 
functions  or  to  be  able  to  address  multiple,  simultaneous  processes. 

While  most  of  the  sources  of  information  comprising  the  different  views  are  external  to  the  controller, 
for  instance  sets  of  local  and  remote  sensors,  some  information  comes  from  the  controlling  system  itself. 
The  regulating  layer  needs  information  about  what  happens  at  the  tracking  layer,  the  monitoring  layer  needs 
information  about  what  happens  -  and  has  happened  -  at  the  regulating  layer,  and  so  on.  Even  a  partial  loss  of 
information  may  lead  to  a  degradation  of  control  and  possibly  destabilise  the  dynamic  equilibrium  among  the 
different  layers.  Since  automation  often  leads  to  a  loss  of  information,  the  principle  of  layers  of  control 
described  by  the  ECOM  provides  a  potentially  powerful  tool  for  analysing  and  understanding  how  such 
effects  may  arise  and  how  they  may  affect  how  well  the  controlling  system  performs. 

7.7.6  Conclusions 

A  first  analysis  of  the  selected  UAV  scenario  shows  that  it  comprises  a  number  of  common  and  specific  tasks. 
The  specific  tasks  can  be  seen  as  constituting  one  line  of  activity,  which  serves  the  primary  purpose  of  the 
scenario,  i.e.,  a  successful  mission.  The  common  tasks  can  be  seen  as  constituting  four  parallel  lines  of 
activity,  which  must  be  carried  out  more  or  less  continuously  in  order  to  ensure  that  the  specific  tasks  can  be 
successfully  accomplished.  As  a  result,  the  scenario  presents  itself  as  comprising  five  parallel  lines  of  activity 
rather  than  just  one. 


7.7.6.1  Effects  of  Automation  on  Common  and  Specific  Tasks 

The  ECOM  characterisation  of  the  common  and  specific  tasks  provides  a  basis  for  considering  the  feasibility 
of  automation  for  various  mission  types.  In  general,  tasks  comprising  regulating  are  the  easiest  to  automate 
from  both  a  technical  and  a  human  factors  point  of  view.  For  an  individual  mission,  this  involves  the 
automation  of  remote  guidance  in  the  sense  that  the  execution  of  actual  manoeuvres  is  done  autonomously. 
This  is  feasible  to  the  extent  that  all  possible  manoeuvres  can  be  anticipated  or  synthesised.  Otherwise  it  falls 
prey  to  the  ‘ironies  of  automation’  [29].  For  a  team/swarm  mission,  regulating  corresponds  to  the  remote 
collective  control  of  the  swarm,  which,  as  noted  above,  can  be  considered  as  one  unit.  Given  that  the 
automation  of  a  single  vehicle  is  possible,  it  should  therefore  also  be  possible  to  automate  the  regulating  of  a 
swarm. 

The  situation  is  somewhat  different  for  a  group,  since  the  individual  members  of  a  group  need  not  behave  in  a 
uniform  manner  (cfi,  the  above  distinction  between  a  group  and  a  swarm  discussed  above).  This  makes  the 
regulating  (as  automated  remote  control)  of  a  group  more  complex  since  it  requires  the  coordinated  regulating 
of  a  number  of  individual  vehicles.  It  may  be  possible  to  the  extent  that  the  coordination  complies  with  a  pre¬ 
defined  pattern  or  patterns,  but  if  that  is  not  the  case,  the  coordination  will  require  targeting  and  monitoring 
functionality  that  makes  it  unsuitable  for  automation. 

Tasks  comprising  monitoring  are  in  general  also  well-suited  for  automation.  Regular  or  routine  monitoring  has 
traditionally  been  one  of  the  earliest  functions  to  succumb  to  automation,  since  it  is  a  function  that  machines 
perform  far  better  than  humans.  In  terms  of  the  descriptions  given  above,  this  applies  to  all  three  types  of 
missions.  Among  the  common  tasks,  monitoring  associated  to  ‘control,  guidance,  navigation’,  ‘system 
management’  and  ‘self  defence’  can  probably  be  automated  for  all  types  as  well.  Among  the  specific  tasks, 
the  same  goes  for  ‘target  registration’,  ‘rules  of  engagement’,  and  ‘battle  damage  assessment’,  but  probably 
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not  for  ‘target  location’,  as  the  latter  may  vary  considerably  with  the  context.  Again,  the  problem  may  be  more 
difficult  for  a  group  type  of  mission,  since  it  involves  differentiated  roles  or  functions  of  individual  vehicles. 

Tasks  comprising  targeting  can,  on  the  whole,  not  easily  be  automated  -  regardless  of  mission  type.  This  is  a 
simple  consequence  of  the  fact  that  automation  is  feasible  only  for  situations  or  conditions  that  have  a  high 
degree  of  regularity  and  predictability.  The  aspirations  and  promises  of  artificial  intelligence  notwithstanding, 
the  take-over  by  technology  of  more  complex  cognitive  functions  has  generally  been  unsuccessful  on  a  real- 
life  scale  (e.g.,  [12]).  For  the  proposed  scenario,  it  is  only  the  common  task  of  ‘self  defence’  and  the  specific 
task  of  ‘re -planning’  that  correspond  to  the  ECOM  targeting  layer.  For  ‘self  defence’,  autonomous  functioning 
may  be  possible  for  common  threat  scenarios,  but  in  that  case  it  may  actually  be  a  question  of  monitoring- 
induced  regulating  rather  than  of  targeting  as  such.  For  ‘re -planning’,  autonomous  functioning  is  probably 
neither  advisable  nor  feasible,  not  even  within  the  time  span  of  the  Roadmap. 

Each  of  the  tasks  can  be  further  characterised  in  terms  of  the  ECOM,  with  respect  to  the  control  layers 
that  are  involved.  This  analysis  shows  that  the  monitoring  and  regulating  layers  dominate.  The  targeting 
layer  occurs  a  few  times,  while  the  tracking  layer  is  assumed  already  to  be  completely  automated. 
The  characterisation  of  tasks  in  terms  of  layers  of  control  provides  a  basis  for  assessing  the  feasibility  of 
automating  specific  tasks  and  functions.  The  general  principle  is  that  regulating  tasks  usually  are  amenable  to 
automation,  while  monitoring  tasks  are  so  to  the  extent  that  they  are  regular  and  predictable.  Tasks  involving 
targeting  are  not  considered  as  likely  candidates  for  automation. 

In  addition  to  considering  each  task  on  its  own,  it  is  also  necessary  to  consider  the  implications  of  having  five 
parallel  lines  of  activity.  In  terms  of  the  ECOM,  this  corresponds  to  having  five  parallel  control  structures. 
In  order  fully  to  appreciate  the  consequences  of  proposed  automation,  it  is  necessary  to  analyse  the 
dependencies  among  the  five  lines  of  activity,  in  terms  of  interacting  goals,  inputs,  and  outputs,  in  addition  to 
doing  so  also  within  each  line  of  activity.  An  analysis  of  this  type  has  not  been  attempted  here,  but  may  be 
carried  out  following  the  same  principles  or  -  if  risk  is  an  issue  -  the  principles  of  functional  resonance  [30]. 


1.1.62  The  Way  Ahead? 

In  the  top-down  approach,  concerns  about  the  level  of  automation  are  secondary  to  the  ability  to  stay  in 
control.  This  means  that  the  discussion  changes  from  a  comparison  of  system  components  and  functions  to  a 
description  of  how  well  the  system  is  able  to  accomplish  its  purpose.  Noting,  for  instance,  that  “the  machine 
suggests  one  alternative  and  executes  that  suggestion  if  human  approves”  does  not  say  very  much  about  the 
nature  of  control  -  except  that  it  must  take  place  at  a  pace  so  slow  that  there  is  time  for  such  consultation. 
A  high  level  of  automation,  or  even  full  automation,  is  not  in  itself  negative  or  bad.  Neither  is  a  low  level  of 
automation  inherently  desirable.  It  all  depends  on  what  the  joint  system  is  required  to  do. 

Describing  system  performance  by  means  of  layers  of  control  makes  it  easier  to  consider  the  consequences  of 
automation,  hence  to  make  sensible  decisions  about  function  allocation.  A  joint  cognitive  system  is  defined  by 
its  ability  to  modify  its  pattern  of  behaviour  on  the  basis  of  past  experience  to  achieve  specific  anti-entropic 
ends,  and  automation  may  affect  this  ability  in  both  a  positive  and  negative  direction.  By  starting  from  an 
understanding  of  control  as  the  ability  to  prevent  unexpected  conditions  from  arising  and  to  recover  from 
them,  should  they  occur,  the  effects  of  design  decisions  are  easier  to  see.  This  goes  both  for  the  dynamic 
equilibrium  among  the  four  different  layers  of  control,  and  the  information  needed  to  sustain  the  views  of  past, 
present  and  future  events.  The  top-down  approach  forces  a  view  of  the  system  as  a  whole,  and  thereby 
weakens  possible  implicit  assumptions  about  what  men  are  better  at  and  what  machines  are  better  at.  Since  it 
is  impossible  completely  to  predetermine  the  environment  in  which  joint  systems  must  function,  every  effort 
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should  be  made  to  ensure  that  they  have  the  capabilities  needed  successfully  to  achieve  their  purpose,  and  to 
maintain  control  even  when  things  do  not  go  as  planned. 
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7.8  SYNTHESIZING  PERSPECTIVE  -  SUPERVISORY  CONTROL  AND 
DECISION  SUPPORT  CONCEPTS 

The  U.S.  Navy  is  depending  very  heavily  on  the  versatile  F/A-18  Super  Hornet  as  the  mainstay  of  its  carrier 
fighter/attack  force  in  the  foreseeable  future.  In  addition,  an  electronic  attack  version  is  also  planned  to 
augment  the  attack  force,  with  deliveries  starting  in  2009.  Aircraft  will  have  either  one  or  two  crewstations 
depending  on  the  version.  On  the  Air  Force  side,  the  F/A-22  Raptor  and  the  F-35  Joint  Strike  Fighter  are  the 
latest  aircraft.  Both  will  have  a  single  person  in  the  crewstation.  The  Navy  and  Marine  Corps  also  plan  to 
purchase  the  F-35.  The  first  deliveries  of  the  Air  Force  and  Marine  Corps  versions  of  the  F-35  will  be  in  2008, 
with  the  Navy’s  first  deliveries  starting  in  2010.  The  bottom  line  is  that  these  three  aircraft  will  provide  the 
two  Services’  fighter/attack  force  well  into  the  future  [1]  -  but  what  type  of  aircraft  will  we  have  beyond 
these?  And  what  type  of  crewstation  will  they  have? 

One  of  the  issues  currently  being  addressed  is  the  role  of  future  long-range  bombers  within  the  Air  Force. 
“The  Air  Force  is  rethinking  long-range  strike,  a  term  that  used  to  mean  only  one  thing:  big  bombers.  As  the 
service  adjusts  to  the  Pentagon’s  new  capabilities-based  strategy  and  focuses  on  desired  effects  rather  than  the 
platforms  needed  to  achieve  them,  the  eventual  successor  to  today’s  bomber  fleet  remains  intentionally 
unsettled.”  [2,  p.  29].  The  various  versions  being  studied  include  not  only  conventional  bombers  as  we  think 
of  them,  but  also  various  types  of  space  planes.  Another  interesting  aspect  of  these  long-range  strike  vehicles 
is  whether  they  will  have  a  crew  onboard  or  on  the  ground.  Among  the  options  being  considered  are  systems 
with  no  airborne  crew  which  means  it  may  become  an  Uninhabited  Aerial  Vehicle  (UAV)  [3]  (The  term 
“uninhabited”  was  chosen  deliberately;  we  think  it  is  more  accurate  than  the  term  “unmanned”  which  implies 
only  a  man  would  not  be  aboard  these  vehicles). 


7.8  .1  Uninhabited  Aerial  Vehicles 

UAVs  have  become  well-known  based  on  the  conflict  in  Afghanistan.  They  served  to  give  the  command  and 
control  authorities  continuous  pictures  of  possible  targets  and  also  enabled  a  dramatic  reduction  in  the  time 
from  which  the  target  was  identified  until  it  could  be  engaged. 

A  number  of  NATO  countries  are  now  using  UAVs  to  augment  their  forces,  especially  in  performing  tasks 
that  are  dull,  dirty,  or  dangerous.  Force  augmentation  issues  relevant  to  the  human  operator  exist  on  several 
levels,  including  individual  UAV  control  station  design,  vehicle  interoperability  by  different  organizations, 
and  integration  of  UAVs  with  manned  systems.  Human  interface  issues  associated  with  individual  UAV 
control  station  design  include  guaranteeing  appropriate  situational  awareness  for  the  task,  minimizing  adverse 
effects  of  lengthy  system  time  delays,  establishing  an  optimum  ratio  of  operators  to  vehicles,  incorporating 
flexible  levels  of  autonomy  (manual  through  semi-autonomous  to  fully  automatic)  and  providing  effective 
information  presentation  and  control  strategies.  UAV  interoperability  requires  development  of  a  standard  set 
of  control  station  design  specifications  and  procedures  to  cover  the  range  of  potential  UAV  operators  and 
applications  across  military  services  and  countries. 

Finally,  for  UAVs  to  be  successful,  they  must  be  fully  integrated  with  manned  systems  so  as  to  enhance  the 
strength  of  the  overall  force.  Human  factors  considerations  in  this  area  include  how  manned  systems  should 
best  collaborate  with  UAVs,  deconfliction  concerns,  operation  with  semiautonomous  systems,  and  command 
and  control  issues.  The  essence  of  this  paragraph  can  be  summarized  by  the  following  statement:  What  is  the 
proper  role  for  the  operator  of  UAVs?  The  operator’s  role  can  be  defined  in  terms  of  three  key  factors. 
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7.8.1. 1  Factor  1:  Advanced  UAV  Operator  Control/Display  Interface  Technologies 

The  operators’  stations  for  the  U.S.  Air  Force’s  Predator  and  Global  Hawk  UAVs  are  mounted  in  vans  with 
the  operators  sitting  at  command  and  control  stations.  The  ground-based  operators  of  these  two  vehicles 
control  them  quite  differently.  The  Predator,  at  least  in  the  landing  in  takeoff  phase,  uses  tele-operation  with 
the  operator  actually  flying  the  vehicle  from  a  distance.  The  Global  Hawk,  on  the  other  hand,  takes  off  and 
lands  automatically  and  is  largely  autonomous  during  its  mission.  The  operator,  using  supervisory  control, 
“flies”  the  Global  Hawk  by  using  a  mouse  and  keyboard,  not  stick  and  throttle.  Not  all  UAV  control  stations 
are  large  enough  to  be  housed  in  vans,  the  operator  station  for  the  U.S.  Marine  Corps’s  Dragon  Eye  UAV, 
for  example,  is  the  size  of  a  small  suitcase  which  makes  it  easily  transportable  (Figure  7-19). 


Figure  7-19:  Predator  Operator  Station  (left)  and  Dragon  Eye  Operator  Station  (right). 

Research  efforts  with  the  Predator  console  have  addressed  a  number  of  control  and  display  features. 
Two  examples  are:  head-coupled  head-mounted  display  applications  [4]  and  tactile  system  alerts  [5]. 
Two  additional  efforts  will  be  discussed  more  in  detail. 

As  an  example  of  a  display  enhancement,  Draper,  et  al.  [6]  examined  four  different  display  formats  which 
would  aid  the  ability  Air  Vehicle  Operator  (AVO)  and  the  Sensor  Operator  (SO)  to  determine  target  location. 
If  the  AVO  located  a  target  in  the  wide  field-of-view  camera,  it  was  often  difficult  to  communicate  the 
location  to  the  SO  who  had  a  narrow  field-of-view  camera.  Four  different  formats  were  examined  to  improve 
communication  between  the  two  crewmembers  (Figure  7-20). 
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Figure  7-20:  Display  Concepts. 
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The  results  showed  that  the  two  formats  utilizing  the  locator  line  were  significantly  better  than  the  others. 
“Time  to  designate  targets  was  reduced  an  average  of  almost  50%  using  the  telestrator  [locator  line]  ...” 
[6,  p.  3-88].  The  reason  for  the  superiority  of  the  locator  line  was  that  once  the  AVO  designated  the  target  it 
gave  the  SO  a  direct  bearing  to  the  target,  thereby  providing  a  very  efficient  means  of  exchanging  information 
between  the  two  operators. 

As  an  example  of  control  research,  Draper  et  al.,  [7]  compared  manual  versus  speech-based  input  involving 
the  use  of  menus  to  complete  data  entry  tasks.  Pilots  also  performed  flight  and  navigation  tasks  in  addition  to 
the  menu  tasks.  Results  showed  that  speech  input  was  significantly  better  than  manual  for  all  eight  different 
kinds  of  data  entry  tasks.  The  overall  reduction  was  approximately  40%  in  task  time  for  voice  entry  when 
compared  with  manual  input.  The  operators  also  rated  manual  input  as  more  difficult  and  imposing  high 
workload  than  the  speech  method.  The  reason  for  the  superiority  of  the  voice  system  was  that  it  enabled  the 
operator  to  go  directly  to  the  proper  command  without  having  to  manually  drill  down  through  a  number  of 
menus  sublevels  in  order  to  find  the  proper  command. 

Different  types  of  control  modes  for  operators’  consoles  were  discussed  in  a  recent  conference  [8]. 
One  recurring  theme  was  a  strong  desire  to  move  away  from  tele-operation  of  the  UAVs  and  progress  towards 
a  combination  of  semiautonomous  and  fully  autonomous  operation  of  these  vehicles  -  regardless  of  the  type 
of  operator  console.  In  order  to  achieve  this  goal,  a  significant  amount  of  automation  will  be  required, 
especially,  when  coupled  with  the  desire,  in  the  case  of  UAVs,  to  move  from  a  situation  where  a  number  of 
operators  control  one  vehicle  to  one  operator  controlling  a  number  of  vehicles. 

Research  exploring  the  issues  of  one  operator  controlling  multiple  vehicles  is  just  beginning.  Barbato, 
Feitshands,  Williams  and  Hughes,  [9]  examined  a  number  of  operator  console  features  that  would  aid  the 
operator  in  controlling  four  Uninhabited  Combat  Aerial  Vehicles  (UCAVs).  The  mission  was  to  carry  out  a 
Suppression  of  Enemy  Air  Defences.  The  operator’s  console  contained  three  liquid  crystal  displays  onto 
which  was  presented  a  situational  awareness  (SA)  map,  UCAV  status  and  multifunction  information.  The  SA 
format  presented  the  overall  geographical  situation  along  with,  among  other  information,  the  flight  routes  of 
the  four  aircraft.  The  participants  were  required  to  manage  the  flight  routes  in  two  ways:  manual  versus 
semiautomatic  using  a  route  planner.  Although  the  operators  were  favorable  towards  the  real-time  route 
planner,  they  did  want  information  regarding  what  the  real-time  planner  was  actually  doing  (its  intent)  and 
they  wanted  both  the  original  route  and  the  planned  route  displayed  in  order  to  evaluate  the  two  against  each 
other.  In  essence,  the  study  showed  that  a  single  operator  could  manage  four  UCAVs  -  so  long  as  there  were 
not  too  many  unexpected  events. 


7.8.1.2  Factor  2:  Supervisory  Control  and  Decision  Support  Concepts 

In  the  case  of  the  UAV,  the  avionics  will  be  partly  contained  in  the  flying  platform  and  partly  incorporated 
into  the  operator’s  console,  whether  airborne  or  ground  based.  In  either  case,  because  of  present  day 
capabilities  in  computers  and  intelligent  agent  software,  the  resulting  product  can  be  much  closer  to  a 
true  team.  Operator-machine  relationships  are  being  created  which  emulate  those  occurring  between  two 
human  crewmembers  -  mutual  support  and  assistance.  A  diagram  depicting  this  overall  relationship  is  shown 
in  Figure  7-21. 
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Figure  7-21:  Operator-UAV  System  Diagram. 


A  major  component  in  achieving  this  mutual  support  and  assistance  is  through  software  entitled  associate 
systems.  “Associate  systems  are  computer-based  aiding  systems  that  are  intended  to  operate  as  an  associate  to 
the  human  user”  [10,  p.  221].  Following  from  his  definition,  Geddes  goes  on  to  list  three  very  important  rules 
for  associate  systems  and  their  relationship  with  the  human  operator. 

•  Mixed  Initiative  -  both  the  human  operator  and  decision  aid  can  take  action. 

•  Bounded  Discretion  -  the  human  operator  is  in  charge. 

•  Domain  Competency  -  decision  aid  has  broad  competency,  but  may  have  less  expertise  than  the 
human  operator. 

Because  of  the  mixed  initiative  aspects  of  an  associate  system,  function  allocation,  which  assigns  roles  to  the 
operator  and  the  computer  based  on  their  abilities,  has  to  be  looked  at  in  an  entirely  new  light.  The  idea  of 
function  allocation  has  been  around  since  the  1950s  and  had  as  its  basic  premise  that  the  role  of  operator  and 
the  machine  (computer),  once  assigned,  would  stay  relatively  constant  during  the  operation  of  the  system. 
However,  this  premise  does  not  hold  for  modern  computers  since  they  contain  associate  systems  which  can 
have  varying  levels  of  automation  at  different  times  during  a  particular  mission;  therefore,  static  function 
allocation  is  no  longer  applicable  [11].  Rather,  dynamic  function  allocation  is  a  key  feature  of  associate 
systems  with  varying  levels  of  automation. 

Taylor  [12],  illustrates  how  dynamic  function  allocation  changes  the  working  relationship  between  the  Human 
Operator  and  the  Machine  (with  associate  system  based  automation);  this  changing  relationship  is  shown  in 
Figure  7-22. 
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Figure  7-22:  Systems  Authority  Concepts  -  H  =  Human;  M  =  Machine  (from  [12,  p.  2-17]. 

Co-operative  Functionings  indicates  how  the  operator  (H  in  the  figure)  and  automation  (M  in  the  figure) 
would  work  together  in  an  associate  system.  It  is  quite  different  from  both  manual  control  and  supervisory 
control.  In  manual  control,  the  human  operator  specifies  the  goals  and  functions  to  be  accomplished  and  the 
machine  carries  out  the  tasks.  In  the  next  level,  supervisory  control,  the  human  operator  still  specifies  the 
goals,  but  the  machine  carries  out  both  the  tasks  and  functions.  In  the  co-operative  functionings  (associate 
system),  the  human  operator  and  machine  interact  at  all  levels,  and  either  can  perform  the  goals,  functions  and 
tasks.  It  is  through  this  dynamic  sharing  of  authority  that  the  operator  and  the  associative  can  begin  to  operate 
as  a  team  -  an  operator  and  a  type  of  electronic  crewmember  (EC).  However,  to  function  as  a  team, 
the  operator  must  trust  the  EC. 

7.8.1.3  Factor  3:  Trust  and  Levels  of  Automation 

One  means  of  establishing  operator  trust  in  the  EC  is  to  allow  the  operator  to  decide  how  much  authority  or 
autonomy,  called  levels  of  automation  (LOA),  to  give  the  EC.  “LOA  defines  a  small  set  (‘levels’)  of  system 
configurations,  each  configuration  specifying  the  degree  of  automation  or  autonomy  (an  ‘operational 
relationship’)  at  which  each  particular  subfunction  performs.  The  pilot  sets  or  resets  the  LOA  to  a  particular 
level  as  a  consequence  of  mission  planning,  anticipated  contingencies,  or  inflight  needs”  [13,  p.  124].  While 
originally  conceived  for  a  piloted  aircraft,  LOAs  apply  equally  well  to  UAV  consoles  and  their  operators.  One 
question  that  must  be  answered  is  how  many  levels  of  automation  should  be  assigned  to  the  associate? 
A  number  of  researchers  have  examined  this  issue.  The  result  is  as  many  as  10  [14]  and  as  few  as  5  [15]. 

In  order  to  create  an  effective  team,  once  the  levels  are  determined,  the  next  task  is  to  determine  how  they 
relate  to  the  way  humans  process  information.  A  further  expansion  of  LOA  was  proposed  by  Parasuraman, 
Sheridan  and  Wickens  [16];  they  matched  levels  of  automation  with  a  four  stage  human  information 
processing  model  (information  acquisition,  information  analysis,  decisions  selection,  and  action 
implementation).  The  10  LOAs  are  based  on  a  model  proposed  by  Sheridan  [14].  They  then  illustrate  how 
various  systems  could  have  different  levels  of  automation  across  the  four  portions  of  the  information 
processing  model.  This  work  is  very  important  because  it  begins  to  blend  levels  of  automation  with  human 
information  processing  capabilities.  The  authors  realize  that  the  model  is  not  finalized,  “We  do  not  claim  that 
our  model  offers  comprehensive  design  principles  but  a  simple  guide.”  (p.  294)  However,  it  certainly  is  in  the 
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right  direction  towards  achieving  an  optimal  matching  between  automation  and  human  capabilities  for 
particular  systems. 

Using  automation  levels  and  having  an  indication  of  the  information  processing  workloads  of  the  mission, 
the  operators  could  establish  a  “contract”  with  the  EC  in  the  pre-mission  phase.  They  could,  through  a 
dialogue  at  a  computer  workstation,  define  what  autonomy  they  wish  the  EC  to  have  as  a  function  of  flight 
phase  and  system  function.  As  an  example,  weapon  consent  would  always  remain  exclusively  the  operator’s 
task,  but  reconfiguration  of  the  UAVs  flight  control  surfaces  to  get  the  best  flight  performance  in  the  event  of 
battle  damage  would  be  the  exclusive  task  of  the  EC. 

7.8.2  Adaptive  Automation 

Although  the  pre-mission  contract  with  the  EC  helps  to  establish  roles  for  it  and  the  human  operator, 
the  functions  allocated  to  each  crewmember  remain  static  throughout  the  mission.  However,  missions  are 
highly  dynamic,  and,  as  stated  before,  it  would  be  desirable  to  change  the  function  allocation  during  the 
mission.  This  dynamic  function  allocation  is  achieved  through  adaptive  automation.  “In  adaptive  automation, 
the  level  or  mode  of  automation  or  the  number  of  systems  that  are  automated  can  be  modified  in  real-time. 
Furthermore,  both  the  human  and  the  machine  share  control  over  changes  and  the  state  of  automation” 
[17,  p.  43]. 

Two  of  the  key  aspects  of  adaptive  automation  are  when  to  trigger  the  shift  and  for  how  long.  The  when  aspect 
is  discussed  by  Scerbo,  Parasuraman,  DiNocera  and  Prinzel  [18,  p.  11]  who  list  a  number  of  methods  for 
triggering  the  shifting  tasks  between  the  operator  and  the  automation:  critical  events,  operator  modeling, 
performance  measurement,  psychophysiological  measurement  and  hybrid  methods.  A  diagram  of  how  many 
of  these  allocation  methods  can  be  used  in  a  system  is  shown  in  Figure  7-23. 


As  an  example  of  how  psychophysiological  measurement  is  used  to  determine  operator  state  Wilson  and 
Russell  [19]  required  U.S.  Air  Force  air  traffic  controllers,  in  a  simulation,  to  manage  air  traffic  around  the 
Los  Angeles  airport.  The  task  loading  was  manipulated  by  the  number  of  aircraft  they  had  to  manage 
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(volume)  and  the  different  kinds  of  aircraft  they  had  to  manage  (complexity).  The  tasks  were  first  given  to 
subject  matter  experts,  and  the  difficulty  was  increased  until  they  verified  that  they  were  in  an  overload 
condition  and  could  not  effectively  handle  the  traffic.  The  participants  were  then  given  the  same  type  of  task 
and  their  physiological  data  was  processed  by  an  artificial  neural  net.  The  result  was  the  neural  net  could 
identify  the  non-overload  condition  99%  of  the  time  and  the  overload  condition  96%  of  the  time.  These  results 
indicate  that  psychophysiological  measures  may  be  potentially  very  useful  in  determining  if  the  operator  is 
overloaded  in  real-world  applications. 

Once  the  state  of  the  operator  can  be  reliably  assessed,  the  next  question  is,  Can  the  workload  be  shifted 
quickly  between  the  operator  and  the  automation?  Wilson,  Lambert  and  Russell  [20]  addressed  this  question 
in  a  study  using  NASA’s  Multi  Attribute  Test  Battery  (MATB).  There  are  four  tasks  in  the  MATB:  tracking, 
systems  monitoring,  resource  management,  and  communications.  As  in  the  air  traffic  control  study  previously 
discussed,  pretest  conditions  were  conducted  to  discover  when  the  operators  were  overloaded,  and  the  neural 
nets  were  used  to  identify  this  condition.  In  one  experimental  condition,  the  participants  managed  all  four  of 
the  tasks,  regardless  of  difficulty.  In  the  other  condition,  when  the  participants  reached  the  overload  condition, 
the  systems  monitoring  and  communications  tasks  were  handed  off  to  the  automation.  The  operator  continued 
controlling  the  tracking  and  resource  management  tasks.  The  results  showed  that,  relative  to  the  manual 
condition,  the  adaptive  aiding  condition  resulted  in  a  44%  reduction  in  tracking  error  and  a  33%  error 
reduction  in  the  resource  management  task. 

The  psychophysiological  triggering  of  adaptation  appears  to  be  very  promising;  however,  researchers  are  still 
very  early  in  applying  this  technology  to  real-world  settings.  “At  present,  however,  there  is  not  enough 
existing  psychophysiological  research  to  provide  adequate  information  on  which  to  base  adaptive  allocation 
decisions”  Prinzel,  Freeman,  Scerbo  and  Mikulka,  [21,  p.  407].  Although  the  shifting  of  tasks  from  the 
operator  to  the  automation  by  psychophysiological  methods  (the  when  aspect)  resulted  in  successful 
performance  in  Wilson  et  al.  [20]  study,  there  doesn’t  appear  to  be  any  general  consensus  as  to  how  long  the 
automation  should  keep  the  transferred  task  in  order  to  optimize  overall  systems  performance.  The  how  long 
aspect  has  been  examined  by  a  number  of  authors,  and  the  answer  appears  to  be  task  specific.  For  example, 
Scallen  and  Hancock  [22]  utilized  adaptive  automation  in  a  study  which  required  pilots  to  perform  tracking, 
monitoring,  and  targeting  tasks  while  flying  a  simulator.  After  a  target  was  presented,  the  tracking  task  was 
automated  for  a  20  second  interval,  after  which  it  was  returned  to  the  pilot.  Conversely,  in  another  research 
effort  [23]  which  looked  at  three  different  cycle  times  between  the  operator  and  the  automation  (15,  30, 
or  60  seconds),  the  15  second  switching  time  resulted  in  the  best  tracking  performance.  However,  three  of  the 
five  pilots  who  took  part  in  the  study  reported  that  the  switching  back  and  forth  was  distracting.  As  a  result, 
the  author  states  that,  “In  the  case  of  adaptive  allocation  systems  we  propose  a  moratorium  strategy  in  which 
there  is  a  minimum  frequency  with  which  the  system  can  either  assume  or  relinquish  task  control.”  (p.  402) 

7.8.3  Putting  It  Together 

Things  are  getting  complicated.  We  now  have  levels  of  automation,  human  information  processing  models, 
and  adaptive  automation.  How  do  we  make  sense  of  all  this?  Kaber,  Prinzel,  Wright,  and  Claman  [24] 
addressed  two  of  the  three  components  in  a  study  which  looked  at  the  issue  of  adaptive  automation  (AA) 
relative  to  the  four  stages  of  the  information  processing  model.  Besides  a  manual  control  condition  where 
there  was  no  AA,  it  was  applied  to  the  all  the  stages  of  the  four  stage  model:  information  acquisition, 
information  analysis,  decision  making,  and  action  implementation. 

The  participants  used  Multitask  which  created  a  simulated  Air  Traffic  Control  environment.  Their  task  was  to 
provide  a  landing  clearance  to  various  aircraft  depicted  on  the  radar  scope.  The  aircraft  were  flying  from  the 
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periphery  to  the  center  of  the  display.  An  error  occurred  if  the  aircraft  reached  the  center  of  the  display, 
or  collided  with  another  aircraft,  before  the  clearance  was  issued.  A  gauge  monitoring  secondary  task  was  also 
used.  If  the  participant’s  performance  on  the  secondary  task  fell  below  a  predetermined  level,  the  primary  task 
would  be  automated.  NASA’s  Task  Load  Index  (TLX)  was  used  to  measure  workload. 

Although  performance  utilizing  AA  was  superior  to  the  manual  control  condition,  the  results  showed  that  AA 
was  most  effective  when  applied  to  the  information  acquisition  and  action  implementation  information 
processing  stages.  It  was  not  effective  in  the  information  analysis  and  decision  making  stages.  The  authors 
conclude,  “All  these  results  suggest  that  humans  are  better  able  to  adapt  to  AA  when  applied  to  lower-level 
sensory  and  psychomotor  functions,  such  as  information  acquisition  and  action  implementation,  as  compared 
to  AA  applied  to  cognitive  (analysis  and  decision  making)  tasks”  [24,  p.  23]. 

The  Kaber  et  al.,  [24]  study  began  to  give  some  insight  into  the  interaction  of  two  components:  information 
processing  and  adaptive  automation  -  but  as  mentioned  at  the  beginning  of  this  section,  there  are  three 
components,  the  third  being  levels  of  automation.  How  do  they  all  fit  together?  Kaber  and  Endsley  [25] 
attempted  to  show  the  relationship  among  all  three  factors.  They  constructed  10  levels  of  automation  and  an 
information  processing  model  similar  to  Parasuraman  et  al.,  [16],  with  the  stages  being  monitoring, 
generating,  selecting,  and  implementing.  In  addition,  they  also  incorporated  adaptive  automation. 

They  then  conducted  a  study  utilizing  six  levels  of  automation:  manual,  action  support,  batch  processing, 
decision  support,  supervisory  control,  and  full  automation  (numbers  1,  2,  3,  5,  9,  and  10  in  Figure  7-24). 
Manual  and  Full  Automation  are  self-explanatory.  Action  Support  is  similar  to  tele-operation.  Batch 
Processing  requires  the  human  to  create  and  decide  the  options  to  implement  and  the  computer  carries  these 
out.  Decision  Support  involves  the  computer’s  suggesting  options  and  once  the  operator  selects  one  of  these 
options  (or  one  self  generated),  it  is  then  put  into  operation  by  the  computer.  In  Supervisory  Control  the 
computer  generates  and  carries  out  the  options.  The  operator  monitors  and  gets  involved  if  necessary. 
These  six  levels  were  then  combined  with  three  levels  of  adaptive  automation  cycle  time  (AACT)  (20%,  40% 
and  60%). 
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Figure  7-24:  LOA  Taxonomy  for  Human-Computer  Performance 
in  Dynamic,  Multi-Task  Scenarios  [25,  p.  119]. 


For  example,  in  a  20  minute  trial  the  task  would  be  allocated  to  the  automation  either  4,  8  or  12  minutes. 
The  results  showed  that,  “The  best  combination  of  LOA  and  AACT  involved  human  strategizing  combined 
with  computer  implementation  (Batch  processing  (LOA  3))  during  high  automation  cycle  times  (12  min  on 
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cycle  and  8-min  off  cycle)”  [25,  p.  147].  This  result  is  a  big  step  forward,  but  also  illustrates  the  difficulty  in 
implementing  adaptive  automation,  levels  of  automation,  and  human  information  processing.  If  we  put  this 
research  on  a  time  scale  relative  to  the  over  80  years  of  research  in  the  design  of  aircraft  crewstations,  we  are 
just  beginning  to  explore  this  area.  So,  we  cannot  expect  instant  answers  to  these  very  difficult  questions. 
To  make  matters  even  more  interesting,  there  are  also  plans  to  place  varying  levels  of  automation  within  the 
airborne  platform. 

7.8.4  Levels  of  Automation  Within  the  Air  Vehicle 

Earlier  in  this  section  it  was  mentioned  that  there  would  be  intelligent  software  both  in  the  operator’s  console 
as  well  as  within  the  UAV  itself.  The  airborne  computing  system  enables  varying  levels  of  autonomy  called 
autonomous  control  levels  (ACLs)  within  the  UAV.  Ten  different  levels  are  shown  in  Figure  7-25.  At  first 
glance,  it  would  seem  logical  to  assume  that  these  10  levels  map  onto  Sheridan’s  10  levels  of  autonomy 
mentioned  in  Factor  3:  Trust  and  Levels  of  Automation.  Sheridan’s  levels  deal  with  the  interaction  between 
the  operator  and  the  UAV.  However,  these  ACLs  are  referring  to  autonomy  levels  within  the  aircraft  only  and 
not  between  the  aircraft  and  the  operator.  One  of  the  interesting  things  about  this  chart  is  that  the  lower  levels 
of  the  chart  refer  to  the  ACLs  within  each  aircraft  in,  for  example,  a  flight  of  four  -  but  from  levels  five 
and  higher,  they  referred  to  how  the  entire  flight  works  together  as  a  group.  The  ten  levels  of  autonomy  in 
Figure  7-25  range  from  Level  1:  Remotely  Guided  (tele-operation)  to  Level  10:  Fully  Autonomous  Swarms 
where  the  vehicles  are  acting  in  concert  with  one  another  to  achieve  a  common  goal. 


Autonomous  Control  Levels 


Figure  7-25:  Aircraft  Control  Levels  [from  26,  p.  84]. 

Tele-operation  has  already  been  discussed  in  Factor  1:  Advanced  UAV  Operator  Control/Display  Interface 
Technologies  and  will  not  be  further  enumerated  upon  here  -  but  Level  10:  Swarms,  which  offer  a  whole  new 
level  of  control  both  within  a  group  of  aircraft  and  between  that  group  and  the  operator,  will  be  examined  in 
more  detail.  The  interesting  thing  about  swarms  is  that  there  doesn’t  appear  to  be  any  central  controller  telling 
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the  swarm  what  to  do.  If  you  observe  a  school  (swarm)  of  fish,  they  just  appear  to  act  as  one  with  no  central 
leader  fish  giving  them  directions.  The  same  is  true  for  flocks  of  birds,  groups  of  ants,  and  swarms  of  bees. 
66  ‘Swarming’  itself  is  a  type  of  emergent  behavior,  a  behavior  that  isn’t  explicitly  programmed,  but  results  as 
a  natural  interaction  of  multiple  entities”  [27,  p.  1].  As  an  example  of  forming  a  swarm,  consider  how  ants 
communicate  that  they  have  found  a  source  of  food.  What  happens  is  that  the  ants  lay  down  a  pheromone  trail 
(chemical  markers)  that  other  ants  can  follow.  The  strength  of  the  pheromones,  however,  decays  over  time; 
therefore,  the  ant  that  finds  the  closest  food  supply  and  returns  with  it  will  have  the  strongest  pheromone  trail. 
Other  ants  will  then  follow  this  trail  with  no  central  commander  ant  directing  them  to  do  this  [28]. 

So,  what  does  this  have  to  do  with  UAVs?  If  a  flight  of  UAVs  could  act  as  a  swarm,  instead  of  giving  them 
explicit,  detailed  instructions  on  the  location  of  surface-to-air  missile  batteries,  for  example,  they  could  be 
directed  to  just  loiter  about  a  certain  area  of  enemy  territory  and  if  they  come  across  the  missiles  they  could 
destroy  them.  Of  course,  they  would  be  acting  within  the  level  of  responsibility  given  to  them  by  the  human 
operator.  Creating  digital  pheromones  for  UAVs  is  one  way  they  could  communicate.  These  types  of 
pheromones  are  not  based  on  chemicals,  but  rather  on  the  strength  of  electrical  fields.  In  a  computer-based 
(constructive)  simulation,  a  UAV  swarm  using  digital  pheromones  significantly  outperformed  the  non-swarm 
case  [29]. 

7.8.5  Conclusion 

UAVs  have  a  wide  range  of  avionics  sophistication,  from  the  relatively  basic  Dragon  Eye  to  very  complex 
Global  Hawks  and  UCAVs.  Many  of  the  UAVs  used  at  the  small  unit  level  will  have  limited  automation 
although,  for  example,  they  will  be  able  to  plan  their  own  flight  route.  However,  most  future  aircraft,  whether 
inhabited  or  not,  will  contain  associate  systems  that  will  incorporate  varying  levels  of  autonomy  and  adaptive 
automation  as  basic  operating  principles.  These  principles  will  enable  the  UAV  operator  and  the  associate  to 
form  a  team  consisting  of  two  crewmembers  -  one  human  and  one  electronic.  In  order  to  function  effectively, 
the  operator  and  the  EC  must  work  together  as  a  close-knit  team,  and  the  EC  may  not  only  supervise  one 
aircraft,  but  the  entire  swarm.  One  essential  feature  of  a  successful  team  is  trust  in  the  other  partner.  As  an 
example,  guidelines  to  create  such  trust  must  include  specifying  the  EC’s  level  of  autonomy.  By  using  these 
guidelines,  a  high-quality,  trusting  relationship  can  be  achieved  between  the  operator  and  the  EC.  This  internal 
trust  will,  in  turn,  lead  to  an  efficient  and  effective  team  which  can  operate  successfully  in  a  system  of  systems 
environment. 
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Chapter  8  -  CONCLUSIONS  AND  SUMMARY 

Chapter  Lead:  J.  Reising 
Contributors:  J  Reising,  R.  Taylor 


Force  augmentation  issues  relevant  to  the  human  operator  have  been  shown  to  exist  on  several  levels, 
including  individual  UMV  control  station  design,  vehicle  interoperability,  and  integration  of  UMVs  with 
manned  systems.  On  the  basis  of  the  material  reviewed  by  the  RTO  HFM-078  Technical  Team,  and  reported 
in  detail  in  the  preceding  chapters,  many  topics,  issues  and  conclusions  related  to  UMV  human  factors  have 
been  raised.  In  an  attempt  to  summarize  this  vast  effort,  five  major  issues  were  extracted  that  cut  across 
several  sections  of  this  report.  Each  issue  is  followed  by  a  representative  sample  of  conclusions  associated 
with  it.  These  identified  conclusions  are  not  at  all  intended  to  be  exhaustive.  Rather  they  serve  primarily  to 
illustrate  the  general  findings  and  understanding  with  regards  to  each  issue. 


8.1  ISSUE  1 :  HUMAN  AUTHORITY  AND  RESPONSIBILITY  IN  DEALING  WITH 
UMVs 

In  modem  asymmetric  warfare,  well-organized  belligerents  ignore  the  legal  requirement  under  international 
law  to  be  readily  distinguished  from  the  civilian  population.  They  merge  with  the  civilian  population,  they  do 
not  travel  in  identifiable  military  vehicles  and  they  use  sophisticated  deception  tactics.  Thus,  in  modern 
warfare,  it  is  very  difficult  for  an  autonomous  machine  to  discriminate  between  civilians  and  military  targets. 

8.1.1  Conclusion  1 

Experienced  human  perception  and  judgment  are  needed  to  assess  risks,  to  consider  both  the  immediate  and 
broader  context,  to  judge  the  consequences  and  implications  of  action,  and  if  possible,  to  anticipate, 
see  through  and  counter  any  new  deception  tactics.  Consequently,  any  autonomous  system  will  remain 
dependent  upon  ‘human-in-the-loop’  targeting  decisions,  where  a  human  makes  the  ultimate  decision  to 
engage  a  target. 

8.1.2  Conclusion  2 

Human  involvement  is  required  in  military  operations  to  direct  and  plan  the  use  of  military  capability,  and  to 
ensure  lawfully  correct  use  of  lethal  force.  This  is  achieved  through  the  application  of  human  command 
authority,  responsibility  and  accountability,  and  competency.  With  autonomous  UMVs,  some  of  that 
responsibility  is  delegated  to  increasingly  competent  computer  controlled  machines,  but  the  authority  and 
accountability  for  the  delegation  ultimately  remains  with  humans. 

8.1.3  Conclusion  3 

Methods  for  expressing  detailed  system  automation  level  requirements  (e.g.,  Pilot  Authorisation  and  Control 
of  Tasks  (PACT))  maintain  operators’  authority  by  enabling  them  to  delegate  responsibility  for  tasks  to  the 
computer  through  a  set  of  contracts  that  limit  autonomy  and  bound  the  behavior  of  the  aiding  system,  while 
maintaining  the  operators’  authority  through  executive  control. 
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8.1.4  Conclusion  4 

Delegation  approaches  to  interaction  with  intelligent  yet  subordinate  human  operators  have  worked  repeatedly 
throughout  history  and,  particularly,  the  history  of  warfare.  Automation  in  the  form  of  UMVs  will 
increasingly  take  its  place  as  one  of  those  actors.  Since  we  want  it  to  be  intelligent,  capable  and  effective, 
yet  remain  subordinate,  we  will  increasingly  need  methods  for  enabling  it  to  interact  with  us  in  the  ways  that 
we  trust  and  are  familiar  with.  Since  delegation  is  the  primary  method  that  fits  that  bill,  it  only  makes  sense  to 
pursue  delegation  approaches  to  human  interaction  with  automation. 


8.2  ISSUE  2:  THE  ROLE  OF  HUMAN  OPERATORS  WITH  ADVANCED 
AUTOMATED  AND  INTELLIGENT  UMV  SYSTEMS 

A  number  of  fundamental  questions  and  key  issues  can  be  identified  concerning  the  role  of  humans  in  advanced 
automated  and  intelligent  systems.  There  is  an  inexorable  trade  off  between  higher  levels  of  automation  and 
unpredictability.  In  particular,  there  is  uncertainty  over  how  to  optimize  the  use  of  human  and  computer  decision 
resources,  while  preserving  a  human-centric  system. 

8.2.1  Conclusion  1 

Automation  must  be  designed  to  augment,  not  hinder,  human  capabilities.  It  is  critical  for  appropriate  use  of 
automation  that  the  user  understand  how  the  automation  works  and  what  mode  the  automation  is  in. 
Additionally,  operator  interfaces  must  provide  rapid  visibility  into  the  current  status  and  future  plans  of 
automation  for  shared  human-automation  situational  awareness. 

8.2.2  Conclusion  2 

Intelligent  decision  support  interfaces  will  need  to  be  designed  such  as  to  allow  independent  operator 
assessment  of  the  situation  as  well  as  the  rationale  for  any  automated  classifications/recommendations. 

8.2.3  Conclusion  3 

The  system  should  perform  automatic  activities  as  if  they  were  completed  by  the  operators  themselves  during 
automatic  task  execution.  There  will  be  less  unattended  actions  of  the  system,  which  improves  the  operators’ 
awareness  and  comfort,  increasing  total  system  safety  and  performance.  Natural  operation  is  particularly 
important  when  the  operators  have  to  override  the  automatic  system  by  switching  back  to  manual  control. 

8.2.4  Conclusion  4 

Automation  does  not  reduce  operator  workload  per  se;  it  may  change  the  nature  of  the  workload  or  may  even 
increase  it.  The  operators,  as  supervisors  of  automated  systems,  essentially  become  long-term  monitors  with 
periodic  requirements  to  intervene  when  necessary.  The  cognitive  workload  associated  with  this  supervisory 
control  may  well  be  higher  than  the  workload  of  physical  control. 

8.2.5  Conclusion  5 

All  automation  is  not  created  equal.  It  can  be  brittle,  unpredictable,  and  prone  to  bias.  Knowing  about  these 
pitfalls  is  half  the  battle.  A  designer  must  carefully  look  at  where  and  how  the  automation  may  fail  and  ensure 
the  operators  know  the  mission  impact  (if  any)  of  the  failure. 
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8.2.6  Conclusion  6 

Human  knowledge,  experience  and  judgment  provide  unique  capability  to  analyze  safety  risks  and  to  think 
ahead  in  uncertain  and  novel  situations.  The  challenge  is  to  provide  information  and  decision  systems  that 
protect  and  preserve  the  human  operators’  key  role,  and  that  augment  and  enhance  the  operators’  cognition 
rather  than  replace  the  operators  in  complex  decision  making. 

8.2.7  Conclusion  7 

New  approaches  to  the  use  of  automation  propose  adjustable  levels  of  computer  autonomy  with  a  strong 
socio-technical  and  cognition  basis.  These  seem  likely  to  provide  sensible  architectures  for  distributed, 
multi-agent  intelligent  systems  that  can  be  more  readily  appreciated  by  human  operators  than  traditional 
automation  approaches. 

8.2.8  Conclusion  8 

Automation  has  often  been  approached  from  the  bottom  up,  starting  with  the  system  components. 
An  alternative  is  to  approach  the  problem  from  the  top  down,  using  the  requirements  to  joint  system 
performance  as  a  starting  point.  In  this  approach  the  emphasis  is  on  operators  being  in  control.  A  multi¬ 
layered  Extended  Control  Model  (ECOM)  provides  a  good  basis  for  understanding  the  consequences  of 
automation  and  the  needs  of  various  types  of  information  to  support  views  of  the  past,  present,  and  future. 


8.3  ISSUE  3:  INTEROPERABILITY  OF  UMV  SYSTEMS 

Migration  of  operator  control  is  currently  regarded  as  one  of  the  most  complex  and  risky  phases  of  UMV 
operations.  Because  it  includes  changes  in  the  locus  of  control  within  functional,  temporal,  or  physical 
domains,  many  system  parameters  may  be  changed  and  difficult  procedural  and  technical  issues  can  be 
involved.  For  instance,  in  current  long  endurance  UAV  operations,  control  may  be  transferred  between 
operators  in  a  control  station  (e.g.,  crew  changeover),  between  control  stations  (e.g.,  vehicle  handoff), 
or  among  members  of  a  crew  (e.g.,  task  execution).  Migrating  control  between  dissimilar  systems  is 
particularly  difficult  because  of  issues  of  system  synchronization. 

8.3.1  Conclusion  1 

The  control  system  will  need  to  be  designed  to  allow  for  system  synchronization  and  facilitate  operators’ 
achieving  an  adequate  level  of  situational  and  system’s  awareness  so  a  handover  can  be  safely  performed. 

8.3.2  Conclusion  2 

UAV  interoperability  requires  development  of  a  standard  set  of  control  station  design  specifications  and 
procedures  to  cover  the  range  of  potential  UAV  operators  and  applications  across  military  services  and 
countries. 

8.3.3  Conclusion  3 

Resolving  issues  associated  with  connectivity,  knowledge  and  action  consistency,  and  transfer  of  control  must 
be  addressed  during  the  early  stages  of  systems  engineering  to  ensure  proper  human-centered  development  of 
UMV  systems  within  a  system-of-systems  architecture. 
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8.3.4  Conclusion  4 

Migration  of  control  between  operators  at  physically  dispersed  locations  may  require  initiation  and  alignment 
of  systems,  one  or  more  data  and  communications  links,  and  possibly  even  cryptological  equipment.  It  may 
also  require  coordination  with  external  command  and  control  agencies.  This  situation  may  be  made  more 
complex  if  a  face-to-face  debrief  is  not  possible. 

8.3.5  Conclusion  5 

Migration  of  operator  control  needs  to  be  coordinated  prior  to  the  actual  event.  This  means  the  specific 
procedures  and  information  to  be  exchanged  should  be  identified  during  the  mission  planning  process. 
The  procedures  should  be  available  in  checklist  form  and  should  have  been  previously  validated  to  minimize 
the  unintended  effects  of  operator  input  errors  as  well  as  be  applicable  to  both  nominal  and  off-nominal 
situations. 

8.3.6  Conclusion  6 

Since  migration  of  operator  control  of  UMVs  demands  a  high  level  of  crew  coordination,  all  involved 
personnel  should  have  initial  and  recurrent  proficiency  training  in  control  transfer  procedures  as  well  as  crew 
coordination. 

8.3.7  Conclusion  7 

Team  performance  directly  correlates  with  team  members’  levels  of  situational  awareness  (SA).  Accordingly, 
in  order  to  safely  migrate  operator  control,  it  is  imperative  the  operators  gaining  control  have  at  least  the  same 
level  of  SA  as  the  operators  releasing  control.  Operators  should  strive  for  the  highest  level  of  SA  (e.g.,  level  3 
SA)  prior  to  assuming  control  of  a  UMV.  Level  3  SA  is  defined  as  prediction  of  the  future  status  of  one’s  own 
situation  and  the  surrounding  elements.  SA  may  need  to  be  achieved  at  the  system,  operational,  and  mission 
levels. 


8.4  ISSUE  4:  CONTROL  STATION  DESIGN 

There  is  a  vast  expanse  of  data  that  is  available  to  UMV  operators  in  a  network  centric  environment.  Coupled 
with  the  limitations  of  human  information  processing,  autonomous  UMV  supervisory  control  issues,  and  the 
impact  of  environmental  stressors  on  cognitive  performance,  control  station  designers  face  a  huge  challenge  to 
provide  a  user  centered  design. 

8.4.1  Conclusion  1 

It  is  important  that  any  UMV  operator  interface  design  follow  a  multi-disciplinary  user-centered  design 
process.  The  goal  of  user-centered  design  is  to  ensure  the  final  design  meets  the  users’  needs  and  expectations. 
The  process  of  requirements  definition  (user  profiles,  work  flow,  task  analysis,  and  information  architecture) 
and  repeated  interface  design  development  and  iteration  (through  multiple  usability  assessments  and  formal 
evaluations)  will  increase  the  likelihood  of  obtaining  fully  functional  and  easy-to-use  interfaces. 

8.4.2  Conclusion  2 

It  is  important  to  recognize  the  unique  challenges  levied  upon  the  UMV  operators  including  the  effects  of 
system  time  delays,  bandwidth  limitations  (which  can  be  intermittent),  datalink  degradations/dropouts,  and  the 
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loss  of  multi-sensory  information  often  afforded  to  onboard  operators.  However,  the  physical  separation  of 
crew  from  vehicle  might  also  offer  some  unique  benefits  that  should  be  exploited.  Besides  the  obvious  benefit 
to  crew  safety,  it  is  quite  likely  that  available  bandwidth  and  the  variety  of  available  information  sources 
might  be,  in  certain  cases,  far  greater  for  a  geographically-separated  UMV  crew  than  for  onboard  operators, 
potentially  resulting  in  more  situational  awareness  rather  than  less. 

8.4.3  Conclusion  3 

As  technology  advances,  the  role  of  the  UMV  operators  must  change  as  well.  Therefore,  UMV  operator 
interfaces  must  be  tailored  to  match  the  capabilities  and  limitations  of  the  host  system  and  intended  mission. 
These  operator  interfaces  must  take  into  account  issues  associated  with  automation  management,  including 
vigilance  effects,  brittle/clumsy  automation,  sudden  workload  spikes,  etc. 

8.4.4  Conclusion  4 

In  the  future,  a  new  interface  paradigm  for  controlling  next  generation  UMVs  may  be  required  to  enable  a 
single  supervisor  to  control  multiple  semi-autonomous  UMVs.  Because  these  UMVs  will  have  the  capability 
to  make  certain  higher-order  decisions,  independent  of  operator  input  and  pre-defined  mission  plans,  operators 
will  face  a  new  set  of  challenges.  Specifically,  they  will  be  required  to  rapidly  judge  the  appropriateness  of 
these  decisions  and  assess  their  impact  on  overall  mission  objectives,  priorities,  etc.  Future  operator  interfaces 
will  need  to  be  tailored  for  multi-UMV  control  and  to  allow  the  operator  the  capability  to  easily 
inspect/override  the  autonomous  UMV  decision-making  logic.  These  interfaces  will  also  need  to  provide 
information  fusing/filtering  algorithms,  intelligent  prioritization/cueing  logic,  and  possibly  some  form  of 
adaptive  task  allocation  in  response  to  rapidly  changing  events  and/or  workload  levels. 

8.4.5  Conclusion  5 

The  ‘T’  arrangement  of  the  airspeed,  altitude  and  heading  in  aircraft  cockpits  has  led  to  a  standard 
arrangement  in  manned  aircraft.  This  has  allowed  pilots  to  move  from  one  aircraft  to  another  with  minimum 
levels  of  negative  transfer.  No  such  standards  exist  for  UMV  control  station  design.  This  has  led  to  vastly 
different  designs  by  each  manufacturer  and  the  result  that  operators  must  be  trained  very  specifically  on  each 
platform  control  station,  with  little  or  no  advantage  of  previous  learning.  This  lack  of  standard  design 
components,  at  least  for  fundamental  information,  must  be  addressed  for  UMVs  to  reduce  training  costs, 
logistics  and  operation  errors. 

8.4.6  Conclusion  6 

UMV  operator  interfaces  need  to  be  designed  with  an  understanding  of  where  the  human  information 
processing  bottlenecks  occur  in  a  task  flow.  As  a  result,  the  operator  must  be  given  information  in  a  form  that 
is  easily  perceived,  interpreted,  and  responded  to. 

8.4.7  Conclusion  7 

Since  UMV  operators  are  currently  limited  to  a  reduced  stream  of  sensory  feedback  delivered  almost 
exclusively  through  the  visual  channel,  there  is  reason  to  believe  that  situational  awareness  and  performance 
may  be  improved  through  multi-sensory  interfaces.  These  improvements  might  stem  from  an  increase  in  the 
operators’  sense  of  presence  in  the  remote  environment,  from  increased  information  throughput  provided  by 
multi-sensory  stimulation,  and/or  a  more  intuitive  presentation/control  of  information.  The  result  can  be 
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improved  performance  over  conventional  visual  interfaces.  Technologies  such  as  spatialized  audio, 
haptic/tactile  stimulation  and  speech  recognition  systems  appear  especially  relevant  to  multi-UMV  operations. 

8.4.8  Conclusion  8 

The  sense  of  presence  (i.e.,  “being  there)  is  often  concomitant  with  engagement  on  the  part  of  the  operators, 
and  this  may  be  critical  when  the  operators  take  on  a  supervisory  role  over  semi-autonomous  UMVs.  In  this 
situation,  there  exists  the  potential  that  the  operators  will  ‘fall  out’  of  the  control  loop  and  may  have  difficulty 
re-entering  when  necessary.  Immersion  in  a  virtual  environment  (i.e.,  the  UMV  operator  interface)  may 
facilitate  intuitive  interaction  and  ensure  that  the  operators  remain  engaged  in  the  mission  even  if  not  directly 
flying  the  vehicle. 


8.5  ISSUE  5:  OPERATOR  SELECTION  AND  TRAINING 

UMVs  are  new  technologies  for  most  militaries  around  the  world,  and  potentially  require  new  jobs,  positions, 
occupations,  and  units  to  command  and  control  these  assets.  On  the  other  hand,  militaries  have  similar 
manned  vehicles  with  similar  payloads.  The  personnel  that  operate  these  vehicles  are  highly  skilled  and 
knowledgeable,  and  these  skills  and  knowledge  are  potentially  transferable  to  operating  UMVs.  Moreover, 
if  UMVs  were  highly  “intelligent”  or  “autonomous”  then  perhaps  only  general  skill  and  knowledge  levels 
would  be  required  to  operate  the  vehicles  and  their  payload. 

8.5.1  Conclusion  1 

The  best  way  to  prevent  the  loss  of  operators’  skills  is  to  periodically  give  the  operators  dedicated  training. 
Another  possibility  is  to  require  the  operators  to  perform  skill  critical  tasks  manually  at  certain  times, 
even  though  the  task  may  have  been  allocated  to  the  automated  system. 

8.5.2  Conclusion  2 

Experience  improves  operators’  cognitive  throughput,  allowing  them  to  devote  limited  attentional  resources  to 
future  problems  while  automatically  attending  to  immediate  perceptual  and  motor  tasks. 

8.5.3  Conclusion  3 

Teams  comprising  fundamental  knowledge,  skills,  and  abilities  (KSAs)  are  better  equipped  to  fulfil  mission 
goals.  KSA  requirements  are  not  completely  transferable  from  in-person  teams  to  virtual  (distributed)  teams 
and  vice  versa.  The  densely  computer-mediated  communication  environment  of  the  virtual  realm  requires  a 
heightened  adeptness  at  managing  digital  conflict,  text-intensive  interactions,  and  media  selection. 

8.5.4  Conclusion  4 

Relative  to  virtual  teams,  social  control  is  particularly  valuable  when  the  need  for  sharing  tacit  knowledge 
increases  over  socially  impoverished  channels  of  virtual  communication,  where  conflict  may  escalate  due  to 
teamwork  issues  engendered  by  cultural  difference  in  communication  and  problem-solving  styles  and 
approaches. 
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8.5.5  Conclusion  5 

Teams  need  environments  which  facilitate  efficient  and  effective  command  and  control  information  sharing. 
When  team  members  trust  each  other  and  the  team  infrastructure,  are  educated  about  organizational  structure 
and  processes,  and  understand  information  processing,  fluid  communication  is  enabled. 

8.5.6  Conclusion  6 

Team  members  need  to  quickly  identify  individual  and  team  information  needs,  fulfil  the  needs, 
and  disseminate,  synthesize,  and  integrate  that  knowledge  into  mission  activities.  Consequently,  situational 
awareness  requirements  can  be  addressed  by  supporting  social  networks  with  access  to  databases,  human 
capital,  and  technology. 

8.5.7  Conclusion  7 

The  transfer  of  skills  and  knowledge,  and  the  requirement  for  general  skill  and  knowledge  levels  will 
contribute  to  Force  Multiplication  by  drawing  from  an  existing,  broader  pool  of  people  that  can  operate 
UMVs. 


8.6  SUMMARY 

A  review  of  the  many  issues  identified  throughout  this  report  highlights  two  major  emphasis  areas  for  future 
NATO  RTO  focus.  The  first  area  involves  the  study  of  tools  and  techniques  for  distributive  collaboration/ 
command  and  control  of  UMV  teams.  Issues  of  virtual  teaming,  communication  bandwidth,  collaboration 
methods  and  responsibility/authority  are  a  few  aspects  of  this  important  area.  The  second  area  centers  on  the 
methodologies  and  technologies  to  enable  flexible  human  supervisory  control  of  multiple,  highly-automated 
UMV  assets.  Issues  with  this  key  area  include  human-automation  challenges  and  mitigation  techniques, 
flexible  levels  of  automation,  situation  assessment  and  decision  support  tools  for  human-robot  systems, 
multi-modal  interfaces,  and  anticipatory  support  aids. 
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